This paper introduces the Weather Research and Forecasting Model with
chemistry/Data Assimilation Research Testbed (WRF-Chem/DART) chemical
transport forecasting/data assimilation system together with the assimilation
of

Trace gas retrieval data sets contain (i) large amounts of data with limited
information content per observation, (ii) error covariance
cross-correlations, and (iii) contributions from the retrieval prior profile
that should be removed before assimilation. Those characteristics present
challenges to the assimilation of retrievals. This paper addresses those
challenges by introducing the assimilation of compact phase space
retrievals (CPSRs). CPSRs are obtained by preprocessing retrieval data sets
with an algorithm that (i) compresses the retrieval data, (ii) diagonalizes
the error covariance, and (iii) removes the retrieval prior profile
contribution. Most modern ensemble assimilation algorithms can efficiently
assimilate CPSRs. Our results show that assimilation of MOPITT CO CPSRs
reduced the number of observations (and assimilation computation costs) by

There is increased international interest in chemical weather forecasting
(Kukkonen et al., 2012; MACC-II Final Report, 2014). Such forecasts rely on coupled
forecast model–data assimilation systems that ingest a combination of
remotely sensed and in situ atmospheric composition observations together
with conventional meteorological observations. Generally the remotely sensed
observations come in the form of trace gas retrievals. Examples include
carbon monoxide (CO) total and partial column or profile retrievals from the
Terra/Measurement of Pollution in the Troposphere (MOPITT) and Metop/Infrared
Atmospheric Sounding Interferometer (IASI) instruments. The associated data
sets are characterized by large numbers of observations with limited
information per observation. Such remotely sensed data have been assimilated
in various settings (e.g. Bei et al., 2008; Herron-Thorpe et al., 2012;
Klonecki et al., 2012; Gaubert et al., 2014), but there have been only a
few papers addressing data compression strategies. Two such papers were
Joiner and da Silva (1998) and Migliorini et al. (2008). This article is
inspired by their research and introduces an efficient assimilation strategy
that reduces the number of MOPITT CO retrieval observations by

Joiner and da Silva (1998) first proposed the idea of using information content to reduce the number of retrieval observations. They suggested projecting retrievals onto the eigenvectors of the observation error covariance matrix and zeroing those coefficients with little or no information. Their approach evolved from one-dimensional retrieval algorithms (e.g. Twomey, 1974; Smith and Woolf, 1976; Thompson, 1992).

Retrievals are often obtained using the optimal estimation method of
Rodgers (2000) to obtain solutions to the

Joiner and da Silva (1998) recognized that when the retrieval was strongly
constrained by the retrieval prior profile, the assumptions underlying
null-space filtering were invalid. For such retrievals, they proposed
filtering based on an eigen-decomposition of

Migliorini et al. (2008) noted that the Joiner and DaSilva (1998) filtering
depended on their truncation criteria and was therefore somewhat arbitrary.
They also showed it was possible to achieve similar filtering results with an
alternative approach that used a more well-defined truncation criterion.
Migliorini et al. (2008) rearranged Eq. (1) to obtain

Migliorini et al. (2008) continued to reduce the dimension of

A more recent paper (Migliorini 2012) shows that retrievals can be transformed to represent only the portion of the state that is well constrained by the original radiance measurements when two requirements are satisfied: (i) the radiance observation operator is approximately linear in a region of state space centered on the retrieval and with a radius on the order of the retrieval error, and (ii) the prior information used to constrain the retrieval does not underrepresent the variability of the state. Migliorini (2012) proves that when those conditions are met the assimilation of radiances is equivalent to the assimilation of retrievals. The Migliorini (2012) analysis shows that it is possible to use information from the retrieval algorithm to compress information in the transformed retrievals

In this paper, we propose an approach that achieves results similar to
(i) Migliorini et al. (2008) without needing to approximate the covariance of

We can rewrite Eq. (2) as

Our approach compresses Eq. (3) so that the dimension of the compact phase
space retrieval (CPSR) profile on the left side of Eq. (5) is identical to
the number of independent linear functions of the atmospheric
profile to which the instrument is sensitive. This method is different from
that of Migliorini et al. (2008) because it compresses the quasi-optimal
retrieval observations based on a linear independence analysis and relies on
the assimilation system to decide how much weight to give the observations.
The approach of Migliorini et al. (2008) reduces the number of observations based
on an uncertainty analysis independent of the assimilation system. Our
approach identifies all linearly independent information contained in the QOR
profile (through projection of the QOR profile onto the left non-zero
singular vectors of the averaging kernel). The approach of Migliorini et al. (2008) may (i) discard some linearly independent information because the
left non-zero singular vectors of the observation error covariance are not
necessarily a basis for the space of QORs, and (ii) discard some linearly
independent information through their uncertainty analysis. Finally, our
approach relies on two transforms: (i) a compression transform (based on the
left non-zero singular vectors of the averaging kernel, and (ii) a
diagonalization transform (based on the left non-zero singular vectors of the
compressed observation error covariance). The approach of Migliorini et al. (2008)
uses two diagonalization transforms – the first based on the observation
error covariance and the second based on the transformed forecast error
covariance in observation space. Our diagonalization transform is analogous
to their first diagonalization transform except we apply it to the compressed
observation error covariance, and they apply it to the

The assimilation of CPSRs should produce results similar to the assimilation of QORs except for (i) the effect of assimilation sub-processes like horizontal localization and inflation, and (ii) differences in the observation error due to the CPSR compression transform. The QOR and CPSR observation errors are different because the compression transform projects the errors onto the leading left singular vectors of the averaging kernel and retains only those components that lie in the range of the averaging kernel.

In summary the steps for obtaining CPSRs from trace-gas retrievals are as
follows (assuming the retrieval equation has the same form as Eq. 2):

Obtain the retrieval and retrieval prior profiles, the averaging kernel, and the observation error covariance for a particular horizontal location.

Subtract the retrieval prior term

Perform a SVD of the averaging kernel. Form a transform matrix from the left singular vectors associated with the non-zero singular values. Left multiply the QOR by the transpose of the transform matrix. This yields the truncated QOR profile as defined by Eq. (4).

Left multiply the observation error covariance by the transpose of the transform matrix. Right multiply that matrix product by the transform matrix. This yields the truncated observation error covariance.

Perform a SVD of the truncated observation error covariance. Scale the left singular vectors with the inverse square root of their respective singular values. Left multiply the truncated QOR profile by the transpose of the scaled left singular vector matrix. This yields the CPSR profile as defined by Eq. (5).

As a check, left multiply the truncated observation error covariance by the transpose of the scaled left singular vector matrix. Right multiply that matrix product by the scaled left singular vector matrix. The result should be an identity matrix with rank equal to the number of non-zero CPSRs from the previous step.

Assimilate the non-zero CPSRs with unitary error variance.

The Weather Research Forecasting Model with chemistry/Data Assimilation
Research Testbed (WRF-Chem/DART) system is the WRF-Chem chemical transport
model (

DART (Anderson et al., 2009) is a community resource for ensemble data assimilation (DA) research developed and maintained by the NCAR/Data Assimilation Research Section (DAReS). DART is a flexible software environment for studying the interaction between different assimilation methods, observation platforms, and forecast models. WRF-Chem and DART are state-of-the-art tools for studying the impact of assimilating trace gas retrievals on conventional and chemical weather analyses and forecasts.

We conducted continuous cycling experiments with WRF-Chem/DART for the period
of 00:00 UTC, 1 June 2008 to 00:00 UTC, 1 July 2008 with 6 h cycling
(00:00, 06:00, 12:00, and 18:00 UTC). To facilitate a large number of
experiments, we used a reduced ensemble of 20 members and a horizontal
resolution of 100 km (

We used NCEP Global Forecast System (GFS) 0.5

For the chemistry initial and lateral boundary conditions, we used global
simulations from the NCAR MOZART-4 model. The fire emissions came from the
Fire Inventory from NCAR (FINNv1; Wiedinmyer et al., 2011), and the Model of
Emissions of Gases and Aerosols from Nature (MEGAN; Guenther et al., 2012)
calculated the biogenic emissions as part of the WRF-Chem forecast. The
anthropogenic emissions were based on the US Environmental Protection
Agency's (EPA's) 2005 National Emissions Inventory (NEI-2005). We used or
adapted existing ACOM/WRF-Chem utilities
(

At each cycle time, depending on the experiment we assimilated meteorology and/or chemistry observations with the DART ensemble adjustment Kalman filter (EAKF) and then advanced the analysis ensemble to the next cycle time with WRF-Chem. The 6 h forecast ensemble was then used as the first guess for the next ensemble DA step.

We assimilated conventional meteorological observations and CO trace gas
retrievals from MOPITT. The meteorological observations were NCEP automated
data processing (ADP) upper air and surface observations (PREPBUFR
observations). They included air temperature, sea level pressure, surface
winds, dew point temperature, sea surface temperature, and upper level winds
from various observing platforms. We refer to those observations as the

We also assimilated MOPITT partial column/profile CO retrievals. MOPITT is an
instrument flying on NASA's Earth Observing System Terra spacecraft. MOPITT's
spatial resolution is 22 km at nadir, and it sees the earth in 640 km wide
swaths. MOPITT uses gas correlation spectroscopy to measure CO in a
thermal-infrared (TIR) band near 4.7

Summary of the WRF-Chem/DART Forecast/Data Assimilation Experiments.

The retrieval error covariance

We conducted two basic experiments: (i) a control experiment where we assimilated only MET OBS (MET DA); and (ii) a chemical data assimilation experiment where we assimilated MET OBS and MOPITT CO partial column retrievals in the form of QORs (MOP QOR). In addition we conducted an experiment where we converted the CHEM OBS to CPSRs and assimilated the CPSRs (MOP CPSR). We also conducted sensitivity experiments where we (i) zeroed the observation error covariance cross-correlations (MOP NROT) – as opposed to using a SVD transformation for diagonalization, and (ii) applied vertical localization (MOP LOC). The suite of experiments is summarized in Table 1.

For all experiments we used (i) DART horizontal and vertical localization – Gaspari–Cohn localization with a localization radius half-width of 3000 km in the horizontal and 8 km in the vertical, (ii) DART prior adaptive inflation, (iii) no posterior inflation, (iv) full interaction between all observations and all state variables – i.e., MET OBS update chemistry state variables and CHEM OBS update meteorology state variables (joint assimilation of MET and CHEM OBS), (v) DART clamping (i.e., the imposition of a minimum threshold) on chemistry state variables to constrain the posterior ensemble members to be positive, and (vi) the reported MOPITT retrieval error covariance as the observation error covariance to account for unrepresented error sources such as representativeness error.

For the MOP QOR experiment, the MOPITT CO retrievals were converted to QORs using an algorithm similar to that described by Migliorini et al. (2008) except we did not perform their second forecast error covariance-based filtering.

The MET DA and MOP QOR experiments are intended to identify the impact of
assimilating chemistry observations. Figure 1a shows shaded contours of CO in
parts per billion (ppb) at

Figure 1b shows the assimilated CO retrievals for the 18:00 UTC,
28 June 2008 update cycle in the upper panel and the corresponding increments
in the lower panel. Comparison of those panels shows that the assimilation
step adjusted the CO concentrations primarily along the satellite observation
paths, which is a consequence of assimilating sparse observations. The DA
adjustments in Fig. 1b are generally consistent with the differences between
MOP QORs and MET DA in Fig. 1a (CO increases in nonpolluted areas – east of
San Francisco, the southeast USA, and Baja). However, that is a general
statement because the MOP QOR – MET DA differences are partially related to
the impact of assimilating CO observations during the preceding assimilation
cycle and partially related to the impact of assimilating all the CO
observations since the beginning of the cycling experiment
(

Figure 2 shows time series of the domain average CO from the MET DA and MOP
QOR experiments in retrieval and state space. The dots represent the
retrieval space results where the cool colors (blue and black) show the
forecasts, and the warm colors (red and magenta) show the analyses. The green
dots represent the MOPITT retrievals. The solid lines show state-space
results. Figure 2 has several interesting results. First, MET DA had a
negative bias of

Time series of the domain average CO from the MOP QOR and MET DA experiments. The red and magenta dots show the domain average CO in retrieval space for the MOP QOR and MET DA analyses denoted in the legend by “A”. The blue and black dots show the domain average CO in retrieval space for the MOP QOR and MET DA forecasts denoted in the legend by “F”. The green dots show the domain average MOPITT CO retrievals. They are the same in both panels and are included for reference. The solid lines show the domain average CO in model space with the same color scheme as used for the analyses and forecasts in retrieval space. The solid lines are the same in both panels are also included for reference.

Figure 3 shows vertical profiles of the time (00:00 UTC, 25 June 2008 to
00:00 UTC, 29 June 2008) and horizontal domain average CO in retrieval
space. It shows that the MOPITT profile had greater vertical variability
(moderate CO near the surface, low CO in the middle troposphere: 500–400 hPa,
high CO in the upper troposphere: 300–200 hPa, and low CO near the
tropopause: 200–100 hPa) than the MET DA and MOP QOR profiles. It also
shows that the assimilation of MOPITT CO had positive impacts throughout the
troposphere with the greatest improvement in the upper troposphere. Figure 3
shows that there were differences in the MOP QOR/MET DA bias reduction
between: (i) the upper and lower troposphere (greater magnitude negative bias
reduction in the upper troposphere and lesser magnitude positive bias
reduction in the lower troposphere), and (ii) the forecast and the analysis
(greater bias reduction in the analysis than in the forecast). Those results
expand our understanding of the bias in Figs. 1 and 2. In Fig. 3 the forecast
and analysis show greater bias reduction in the upper troposphere. That
suggests that the domain averages in Fig. 2 were dominated by bias reductions
in the upper troposphere. Figure 3 also suggests that bias reductions in the
lower troposphere were dominated by the reduction of the positive bias in the
polluted areas of Fig. 1. Those results suggest that the following model
errors (as opposed to initial/boundary condition errors) caused the biases:
(i) the near-surface biases were likely caused by the CO emissions being too
high in polluted areas and too low in nonpolluted areas, (ii) the positive
biases in the lower middle troposphere (

Vertical profiles of time/horizontal domain average CO from the MOP
QOR and MET DA experiments for 00:00 UTC, 25–29 June 2008. The results are
in MOPTT retrieval space. The red profiles represent the MOPITT retrievals.
Otherwise the color of the lines corresponds to the legend.

Lastly, we tested the null hypothesis that the difference between the MET DA
and MOP QOR time series results was zero (H0: MOP QOR

Next we study the assimilation of CPSRs as described in Sect. 2 but first
review some CPSR attributes. Figure 4a shows vertical profiles of CPSR
characteristics averaged for the MOPITT retrieval domain at 18:00 UTC,
28 June 2008. The blue curves represent the MOPITT CO retrievals (MOP-Rets).
Those curves have reduced vertical structure due to the units
(log

In Fig. 4a the residuals are defined as the difference between the full and
truncated profiles (e.g.,

In summary Fig. 4a shows the state space impacts from applying the Migliorini et al. (2008) quasi-optimal subtraction and the CPSR transform in (i). It also shows that the quasi-optimal subtraction was necessary to remove the influence of the retrieval prior. Thus, in CPSRs the quasi-optimal subtraction removes the influence of the retrieval prior, and projection onto the leading singular vectors of the averaging kernel provides the data compression.

In Fig. 4a the average number of leading singular vectors was

In Fig. 4b we examine the vertical structure of the CPSRs. The upper panel shows the retrieval domain average of the MOPITT averaging kernel profiles from 18:00 UTC, 28 June 2008. The lower panel shows the domain average of the leading left singular vectors from SVDs of the averaging kernel profiles in the upper panel. Comparison of the upper and lower panels shows that while the singular vector and averaging kernel profiles are similar it is not possible to associate a specific singular vector profile with a specific averaging kernel profile or with a group of profiles. However, the averaging kernel of singular vectors show the sensitivity of the associated CPSR to the true CO profile. The first singular vector shows positive sensitivity to the entire CO profile with greater sensitivity in the lower and middle troposphere and greatest sensitivity in the upper middle troposphere. The second singular vector shows positive sensitivity in the lower troposphere and negative sensitivity in the upper troposphere. Lastly, the third singular vector resembles the first singular vector with greatest positive sensitivity in the upper middle troposphere but with negative sensitivity in the lower and upper troposphere. Those characteristics are consistent with the MOPITT TIR and NIR joint sensitivities documented by Worden et al. (2010), Deeter (2011), and Deeter et al. (2012, 2013). It should be noted that the sign of the singular vectors in the Fig. 4b is arbitrary because the left and right singular vectors can be jointly multiplied by negative one and still qualify as singular vectors. However, when multiplied by one sign the singular vector may have physical meaning, and when multiplied by the other it may not. For our application, the sign that made the vertical structure of the singular vectors most similar to that of the averaging kernel had physical meaning. Therefore, in Fig. 4b we chose the sign that made the singular vector profile most consistent with the averaging kernel profiles.

To test the benefit of assimilating CPSRs we converted the MOPITT CO
retrievals to CPSRs and repeated the MOP QOR experiment (called MOP CPSR).
Those results are shown in Figs. 5–7. Conceptually the MOP CPSR results
should be similar to the MOP QOR results in Figs. 1–3. Practically, the
results are different due to (i) the effect of DA sub-processes like
horizontal localization and inflation, and (ii) differences in the
observation error caused by the CPSR compression transform. Comparison of the
contour maps in Figs. 1a and 5a shows that MOP CPSR provided similar
adjustments to MOP QOR but they were of greater magnitude and larger area
(the MET DA result was not plotted in Fig. 5a because it would be the same as
in Fig. 1a). The general trend from Fig. 1a that the assimilation of CO
retrievals reduced the positive CO bias in polluted areas and the negative
bias in nonpolluted areas appears in Fig. 5a. Comparison of Figs. 1b and 5b
shows that MOP CPSR generally assimilated the same CO retrievals as MOP QOR,
but the CPSR increments were of greater magnitude and more widely dispersed.
Comparison of the time series plots in Figs. 2 and 6 shows that there were
slightly greater bias reductions for MOP CPSR than MOP QOR. MOP CPSR reduced
the CO negative bias in retrieval space by

Same as Fig. 2 except for the MOP CPSR experiment.

We calculated verification statistics (bias, root mean square error (RMSE),
and coefficient of determination (

Same as Fig. 3 except for the MOP CPSR experiment.

One method used to diagonalize the observation error covariance is zeroing of the cross-correlations (see the Introduction to Migliorini et al., 2008). The uncertainty of the error covariance and the practice of adjusting the observation error variance to tune ensemble DA strategies are used to justify the zeroing. As noted by Anderson (2001) and applied by Migliorini et al. (2008), a more aesthetic and mathematically correct approach is to apply a variance maximizing rotation based on a SVD of the error covariance. In this section we compare those two error covariance diagonalization methods. Recall that MOP QOR used an SVD-based rotation to diagonalize the error covariance. We conducted a companion experiment MOP NROT where we used cross-correlation zeroing. The assimilation/forecast plots are not shown because it is not a central theme of this paper. However, we include the verification statistics in Fig. 8. Significance testing and scores from assimilation of MOPITT CO show that SVD-based diagonalization produced significantly greater forecast skill compared to cross-correlation zeroing. Based on that result we conclude that the second SVD-based rotation is a necessary step in our definition of CPSRs.

Verification statistics for all experiments in MOPITT retrieval space. The blue curve is the bias (model – observation), the red curve is the root mean square error (RMSE), and the magenta curve is the coefficient of determination. The experiments are described in the text and summarized in Table 1.

Ensemble data assimilation generally uses localization to remove spurious
correlations that may occur from under-sampling. Localization limits the
horizontal and vertical spatial scales on which the observations impact the
posterior. Vertical localization may be inappropriate when assimilating phase
space retrievals because

In this paper we incorporated WRF-Chem into DART and assimilated MOPITT CO
trace gas retrievals. We also introduced the assimilation of compact phase
space retrievals (CPSRs). CPSRs are preprocessed trace gas retrievals that
have (i) the influence of the retrieval prior removed, (ii) data
compression, (iii) SVD-based error covariance diagonalization, and (iv) unit
error variance scaling. We showed that assimilation of CPSRs is an efficient
alternative to assimilation of quasi-optimal retrievals (QORs) that provided
substantial reductions in computation time (

We presented results from month-long (00:00 UTC, 1 June 2008 to 18:00 UTC,
31 June 2008) cycling experiments where we assimilated conventional
meteorology and MOPITT CO retrievals. For MOP QOR the time series plots in
Fig. 2 showed that MET DA had a negative bias of

Next we focused on making the assimilation of retrievals computationally efficient and introduced compact phase space retrievals. CPSRs advance the work of Joiner and DaSilva (1998) and Migliorini et al. (2008) by describing an easily applied methodology to achieve data compression for phase space retrievals. Conceptually, the assimilation of QORs and CPSRs should yield similar results except for the effects of (i) assimilation sub-processes like localization and inflation and (ii) different observation errors due to the CPSR compression transform. Nevertheless, our CPSR approach is different from that of Migliorini et al. (2008): (i) we perform two transforms – a compression transform and a diagonaliztion transform, they perform two diagonalization transforms; (ii) we identify and assimilate all linearly independent information observed by the instrument, they may discard linearly independent information – some because their transform vectors are not necessarily a basis for the space of QORs and some because their uncertainty analysis discards some information that lies in the range of their transformed averaging kernel; (iii) our diagonalization transform is analogous to their first diagonalization transform except we diagonalize the compressed observation error covariance and they diagonalize the untransformed observation error covariance; and (iv) we rely on the assimilation system to decide how much weight to give the transformed observations and require no information from the forecast ensemble, and they use the forecast ensemble to decide which observations to discard.

MOP CPSR maps in Fig. 5 showed
that assimilation of CPSRs placed CO hot spots in the same locations as MOP
QOR but they were of greater magnitude and larger area. The time series and
vertical profile plots showed that those differences generally represented
analysis and forecast improvements. Skill metrics for MOP CPSR showed that
when compared to MOP QOR, the assimilation of CPSRs slightly improved the
forecast skill for all metrics, and when compared to MET DA it significantly
improved the forecast skill for all metrics. Based on those results we
conclude that the assimilation of CPSRs performed as well or better than the
assimilation of QORs at a substantially reduced computational cost
(

Collectively our analysis of the MOP QOR and CPSR results in Figs. 1–3 and 5–7 suggested that (i) in the lower troposphere MET DA had a negative CO bias in polluted areas and a positive bias in nonpolluted areas (Figs. 1 and 5) and (ii) bias reductions in the domain average retrieval space CO were due to reductions in the negative CO bias in the upper troposphere (Figs. 2, 3, 6, and 7). We proposed three causes for the CO biases: (i) emission errors – overestimation of CO emissions in polluted areas and underestimation in nonpolluted areas, (ii) transport errors – too much CO transport from the near surface to the lower troposphere and too little transport from the lower to upper troposphere, and (iii) chemistry errors – too little CO destruction in the near surface and lower troposphere and too much destruction in the upper troposphere.

We expect that CPSRs have the potential for broad operational application. CPSRs can be easily obtained from retrievals derived from any optimal estimation algorithm. They can be used to assimilate retrievals with correlated or uncorrelated errors for any sequential assimilation methodology (both Kalman filter and variational-based algorithms). Due to their ease of derivation, flexibility, and potential for large reductions in assimilation computation time, CPSRs should facilitate the efficient assimilation of dense geostationary observations.

NCAR is sponsored by the National Science Foundation (NSF). Any opinions, findings and conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of NSF. This research was also sponsored by NASA grants NNX11A110G and NNX10AH45G. We gratefully acknowledge the anonymous reviewers, Chris Snyder, and Louisa Emmons for their thorough reviews of this manuscript and for providing many constructive comments. We also acknowledge Helen Worden for assistance with the MOPITT data, and Jerome Barre for assistance with coding the WRF-Chem initial and boundary condition perturbations. Edited by: A. Lauer