GEOS-Chem is a three-dimensional CTM driven by assimilated meteorological data from the Goddard Earth Observation System (GEOS) of the NASA Global Modeling and Assimilation Office (GMAO). Two alternative data sets can be used, either the GEOS Fast Processing (GEOS-FP) data at 0.25∘ × 0.3125∘ native resolution or the GEOS Modern-Era Retrospective Analysis for Research and Applications version 2 (MERRA-2) data at 0.5∘ × 0.625∘ native resolution. Both have 1 h temporal resolution for transport (archived winds) and extend from the surface to the mesopause. GEOS-Chem simulates atmospheric species concentrations by solving the coupled 3D Eulerian continuity equations on a global or user-selected nested domain at the native grid resolution of the GEOS data or at degraded resolution for computational economy. Input data files are regridded on the fly to user-specified resolution using the Harmonized Emissions Component (HEMCO) (Keller et al., 2014; Lin et al., 2021). CHEEREIO supports all GEOS-Chem applications from version 13.0.0 and later including oxidant–aerosol chemistry, aerosol only, carbon gases, and mercury, either as global or nested-grid regional simulations. GEOS-Chem High Performance (GCHP) (Eastham et al., 2018; Martin et al., 2022), which uses distributed memory rather than shared memory for parallelization, is not currently supported.

HEMCO is a critical GEOS-Chem module enabling the interface with CHEEREIO. It can apply gridded scaling factors stored in NetCDF files to any input field, such as emissions. This allows emissions updates calculated by CHEEREIO to be seamlessly loaded into GEOS-Chem without modification of source code.

The LETKF algorithm optimizes a state vector of emissions and concentrations to minimize the cost function in Eq. () (Hunt et al., 2007). We initialize m ensemble members at time to and run the forward model (GEOS-Chem) in parallel for a user-specified time (termed the assimilation window) for each of these ensemble members. Ensemble members can be thought of as a Monte Carlo sample representing the spread of atmospheric conditions resulting from our uncertainty in prior emissions; each member represents the atmospheric conditions from a random emissions perturbation sampled from a user-specified PDF. Before assimilation begins, the ensemble is run for a spin-up period to ensure the propagation of information from the perturbed emissions. Alternatively, the problem can also be set up by perturbing concentrations to represent prior uncertainty in the atmospheric state. Ensemble size is typically between 24 and 48, with the exact number to be determined by sensitivity testing, where the user identifies a size that balances error minimization with computational feasibility. Because ensemble methods randomly sample the parameter space, increasing ensemble size gradually yields diminishing return. This is unlike the analytical approach to inversions, where the number of simulations is set by the size of the state vector and diminishing return is defined by the rank of the problem (Nesser et al., 2021). In general, fewer ensemble members are required if there are fewer parameters to optimize (Miyazaki et al., 2012b; Liu et al., 2019). After the runs are complete, we construct the state vectors xib representing the concentrations and/or emissions to be optimized for each of the m ensemble members (indexed by i).

The LETKF algorithm, which we describe in the remainder of this section, is typically applied to very large state vectors, for which global optimization would be computationally prohibitive. The solution of Hunt et al. (2007) is to localize the calculation within a certain radius of the grid cell being optimized, considering only observations within that radius. Localized state vectors are formed by concatenating emissions at a given grid cell with concentrations within a given radius. Beyond reducing state vector size, this approach creates an embarrassingly parallel problem, where the cost function can be minimized independently for every localized point. The Hunt et al. (2007) localization approach also minimizes spurious correlations, which emerge in ensemble approaches due to a limited sample size; because the Monte Carlo sample is far smaller than the dimensionality of the state vector, random points will be spuriously correlated in the prior covariance matrix Pb encoded by the ensemble, leading to an incorrect assimilation increment. The spurious correlation problem is especially pronounced between distant grid cells where we would expect correlations to be near zero, a problem eliminated by appropriate localization. Such localization in space is not generally useful in the analytical inversion approach, where distant correlations are set to zero and observations are ingested sequentially (Brasseur and Jacob, 2017). The precise radius used for localization should be determined by the user via sensitivity tests, considering that longer-lived species require larger localization radii; indeed, within a single inversion, multiple localization radii can be used for different components of the state vector (Miyazaki et al., 2012b). For the remainder of the equations in this section, all vectors and matrices are localized and computations are performed in parallel.

To optimize the emissions and concentrations of a given grid cell, we first construct the ensemble state vectors xib using model data. From these prior state vectors the prior perturbation matrix Xb is formed from the m vector columns Xib. 2Xib=xib-xb‾;xb‾=1m∑i=1mxib

Here Xib represents the ith column of the n × m matrix Xb where n is the length of the state vector; each column of Xb consists of the state vector from an ensemble member minus the mean state vector. The prior covariance matrix Pb can be constructed by multiplying Xb with its transpose (specifically, Pb=(m-1)-1Xb(Xb)T), but this is not used directly in LETKF calculations. The model predictions made during the assimilation window must be compared to observations. Hence we construct prior vectors of simulated observations yib and a corresponding simulated observation perturbation matrix Yb formed from the m vector columns Yib. 3Yib=yib-yb‾;yib=Hxib;yb‾=1m∑i=1myib

4D-LETKF, the method used in CHEEREIO, constructs Yib such that all simulated observations are timed to line up as closely as possible with actual observations (Hunt et al., 2007). 3D assimilation, by contrast, only aligns observations in space but uses a single model state (in particular, the state at assimilation time) to construct Yib, leading to significant representation error. For 4D-LETKF, we load in model history files closest in time to the observation of interest and accept a modest representation error; the user can specify the time resolution via the CHEEREIO configuration file. Hence in practice we apply the operator H(⚫) to the forward model history, not to the state vector which represents the model state at a specific point in time. Methods which make use of the TLM, like 4D-Var and EKF, avoid temporal representation error due to the continuous ingestion of observations on the internal time step of the TLM, but they require major time investment in TLM development and maintenance.

Computation of the cost function in Eq. () involves inversion of the prior error covariance matrix Pb, but this is not possible in the state space (of dimension n) because by construction Pb is of rank m-1 (the columns of the n × m matrix Xb sum to the 0 vector). Hence a posterior error covariance matrix Pa must be estimated in the m-1 dimensional subspace S spanned by the ensemble perturbations, where the inverse is well-defined. The mathematics simplify by treating Xb as a linear transformation from some m-dimensional space S̃ to S, allowing us to redefine the cost function optimization in S̃ where the relevant quantities are well-behaved (Hunt et al., 2007). The posterior error covariance in S̃ noted with a tilde Pã is an m × m matrix computed as follows. 4Pã=(m-1)⋅I1+Δ+γ(Yb)TR-1Yb-1

The full derivation of Pã is given in Hunt et al. (2007). Here I is the m × m identity matrix, R is the observational error covariance matrix, and γ is a regularization constant set by the user. Pã plays the same role in LETKF assimilation as the posterior covariance matrix Pa plays in the classical Kalman filter, connecting the state vector entries so that the calculated update is consistent with the internal correlations of the system. The regularization constant effectively scales observational errors and is designed to balance the weight given to observations within the assimilation window. Δ is an inflation factor specified by the user, usually between 0 and 0.1, which accounts for overconfidence in the assimilated ensemble; Δ  does not affect the ensemble mean but it does increase the ensemble spread, with larger values pushing ensemble members away from the ensemble mean. In practice, ensemble spread can decrease with each assimilation cycle to values so small that the system is no longer able to update (nearly infinite confidence is given to the prior term, so subsequent observations carry no weight). Indeed, if γ balances the weight given to observations within the present assimilation window, Δ can be thought of as a term that balances the weight given to observations from all previous assimilation windows.

The mean posterior state vector in the original space is then given by 5xa‾=xb‾+γXbPã(Yb)TR-1y-yb‾, where y is the vector of observations. The posterior perturbation matrix is given by 6Xa=Xb(m-1)Pã12.

From here, the new ensemble state vectors can be constructed by adding xa‾ back to each column of Xa. The LETKF gives error characterization from the assimilation; to obtain this, we need to transform Pã back from the space S̃ to the original state space. Since we defined S̃ with the linear transformation Xb, the posterior error covariance matrix is given by 7Pa=XbPã(Xb)T.

With the ensemble updated and errors characterized, the ensemble can be evolved using GEOS-Chem for the next assimilation window. Importantly, the ensemble is not re-initialized for these new runs; the assimilated state of the previous assimilation window becomes the initial prior state of the next assimilation window. When the runs in the new assimilation window are complete, the whole LETKF cycle begins again.

Many variations of the ensemble Kalman filter algorithms have been developed in the chemical data assimilation literature, each designed to better handle the behaviors of certain atmospheric constituents. For example, the Carbon Tracker CH4 system handles the assimilation of long-lived CH4 via a sliding-window approach, where surface fluxes from a given time period are estimated several times using a varying set of observations that evolve in time (Peters et al., 2005; Bruhwiler et al., 2014). Similarly, the run-in-place method changes the behavior of the assimilation window to better handle long-lived gases like CH4 (Liu et al., 2019). With the run-in-place functionality activated, the LETKF assimilation update is calculated using a long period of observations (e.g., 1 week) but the assimilation window is advanced forward for a smaller amount of time (e.g., 1 d). Run-in-place simulations thus maintain linear growth in posterior perturbations and allow the period in which the assimilation update is calculated to experience the emissions adjustment, giving the system more time to correct assimilation errors. CHEEREIO supports many of these variations on the LETKF, as discussed in Sect. 3.

Figure 1 shows a schematic of the CHEEREIO workflow including initialization, spin-up, sequential GEOS-Chem forward model runs and LETKF assimilation, and post-processing. Here we give a high-level overview of how CHEEREIO can be customized and deployed for any chemical data assimilation applications with GEOS-Chem. In subsequent sections, we will offer more detailed descriptions of the software design and structure, omitting technical details provided in the web documentation (https://cheereio.readthedocs.io, last access: 22 August 2023).

The CHEEREIO software package includes a suite of shell and Python scripts for assimilation, run management, observation operations, and post-processing, which can be separated into three main sequential periods in the CHEEREIO workflow, as shown in Fig. 1: initialization time, runtime, and post-processing time. Before initialization begins, users specify the simulation they would like to run via an ensemble configuration file (ens_config.json). CHEEREIO generates a template GEOS-Chem run directory based on user settings, which is copied into an ensemble of m run directories, one per ensemble member, each with a unique set of emissions perturbed according to user settings (Sect. 3.2). After the ensemble is initialized, the user submits a batch script which launches an ensemble of jobs, each running an instance of GEOS-Chem. Once the model simulation for the assimilation window is complete, CHEEREIO gathers model output files and observation data and performs the LETKF assimilation. The cycle of GEOS-Chem runs and LETKF assimilation repeats until the assimilation is complete for the entire user-specified period. Run management and LETKF implementation are discussed in Sect. 3.3. Upon completion, the user can execute a post-process workflow job to make a default set of figures, movies, and consolidated data files; they can also deploy pre-written functions to produce custom output and statistics (Sect. 3.4). In the coming sections, we expand on each of these components of the CHEEREIO workflow.

The CHEEREIO ensemble initialization workflow is divided into four phases, as shown in Fig. 1: (1) template run directory creation, (2) spin-up of template run directory, (3) ensemble initialization and prior emissions sampling, and (4) spin-up of the ensemble spread.

Initialization begins when the user specifies the simulation they would like to run by modifying a configuration file (ens_config.json), which includes all model and assimilation settings. Table 1 lists important parameters that can be tuned in this configuration file. LETKF results respond strongly to the localization radius (LOCALIZATION_RADIUS_km), the regularization factor γ (regularization_factor_gamma), and the ensemble inflation factor Δ (inflation_factor), all of which modulate the weight given to observations in the assimilation calculation. The assimilation window length (assim_time) governs the timescale of the response of the LETKF system to changes in observations and also strongly influences results. Detailed instructions on all settings are in the online documentation (cheereio.readthedocs.io). CHEEREIO then creates a template GEOS-Chem run directory reflecting user settings, which will eventually be copied into an ensemble of m run directories, one per ensemble member; if users modify the template before the ensemble is created, such as to customize emissions inventories, their adjustments will be reflected in each ensemble member. Hence the template run directory allows users to customize their simulations beyond the parameters available in the ens_config.json configuration file. Like any atmospheric model, CHEEREIO must be spun up before any run begins so that it reflects realistic atmospheric conditions; spinning up the template run directory allows the user to run one universal spin-up simulation for all m ensemble members.

With the template initialized, compiled, and spun up, CHEEREIO copies the template into an ensemble of m run directories. Each ensemble member is differentiated by a unique initial perturbation to user-specified emissions, reflecting prior uncertainty in emissions. For example, users interested in assimilating NO2 observations might specify that they have some prior uncertainty in NOx emissions; CHEEREIO will then initialize unique grids of NOx emissions in each ensemble member run directory by drawing samples from a user-specified PDF that perturb existing emissions inventories. These prior errors can be sampled from a normal distribution and can include spatial correlations specified by a correlation distance.

Users can choose to use emissions sampled from either a normal or lognormal spread. If users opt for lognormal emissions, then CHEEREIO samples multiplicative perturbations from a normal distribution centered on zero and then exponentiates to obtain a lognormally distributed sample with a mode of one (i.e., the prior). To meet the LETKF algorithm assumptions, in the lognormal case emissions are transformed back into a normal distribution during the LETKF calculation, before being exponentiated back to a lognormal for use in GEOS-Chem. Benefits of using a lognormal spread include natural protection against negative scaling factors (the lognormal distribution is positive) and a more realistic representation of high-tail uncertainties in emissions inventories.

CHEEREIO grants wide flexibility to users in how emissions perturbations are defined across the ensemble. Users can group emissions of multiple species together into one consolidated entry in the state vector (e.g., NOx), updated at once at assimilation time. Users can also differentiate emissions by source by separately perturbing subsets of emissions, such as methane from oil and gas and methane from agriculture. The resulting assimilation will provide the user with separate emissions updates for each source, allowing users to easily run source attribution studies. Sectoral separation of emissions is implemented naturally in the LETKF formulation by defining the state vector so that separate source sectors have separate 2D representations; if sources overlap in space, the assimilation update will increment both according to the correlation strength in the prior error covariance matrix, which the user must keep in mind while interpreting source separation results.

Before the assimilation cycle begins, users must run a CHEEREIO-specific spin-up process to create a spread in simulated atmospheric conditions across ensemble members, reflecting the initial perturbations in emissions. Because the LETKF algorithm uses spreads in simulated concentrations across the ensemble to approximate the prior error covariance matrix Pb, the model must be run for some period before assimilation begins in order to ensure that variations in concentrations across ensemble members reflect variations in emissions. If this ensemble-wide spin-up is neglected or run for too short a period, Pb will be too small and observations will be neglected (because they will be weighted negligibly in the cost function).

Figure 2 shows a schematic of the CHEEREIO runtime processes. From a computational cluster perspective, CHEEREIO is an array of m jobs, where m is the number of ensemble members specified by the user; each job is allocated p cores as specified by the user. Each job alternates between running GEOS-Chem for an ensemble member and running assimilation scripts for a subset of grid cells – each parallelized separately across the p cores allocated to each job. In this section, we discuss the implementation and control of this complex of processes.

When the CHEEREIO runtime is process complete, users can execute the post-process batch script to automatically consolidate GEOS-Chem diagnostic and emissions output into NetCDF files, along with a file pairing actual observations with simulated observations from the ensemble. Users can also run a control (“prior”) simulation with no assimilation within the CHEEREIO environment; output from this run is automatically handled by the post-processing utility to produce plots and data that compare assimilated output to control output. CHEEREIO also produces a suite of graphs and animations depicting a variety of output including scaling factors, concentrations, emissions, and observation information. All plots of results in Sect. 4 of this paper are generated by the CHEEREIO post-processing utility with no additional code. To facilitate additional analysis, a post-processing toolkit is provided for user processing of both consolidated output files and raw ensemble output.

For our demonstration, we use the methane simulation from GEOS-Chem version 14.0.2 (10.5281/zenodo.7383492, The International GEOS-Chem User Community, 2022) at 2.0∘ × 2.5∘ spatial resolution and following the setup described by Qu et al. (2021). Methane emissions come from anthropogenic sources including livestock, oil and gas, coal mining, landfills, wastewater, and rice cultivation, as well as from natural sources including wetlands and termites. Loss is primarily due to oxidation by OH, with additional minor terms from oxidation by Cl atoms, stratospheric oxidation, and soil uptake.

Methane observations used for data assimilation are from the TROPOspheric Monitoring Instrument (TROPOMI) scientific product 2.2.0, shown in Fig. 3 (Lorente et al., 2021a). TROPOMI retrieves daily global dry methane column mixing ratios (XCH4) at 5.5 km × 7 km nadir pixel resolution at ∼ 13:30 local solar time. In our demonstration, we filter TROPOMI observations to include only those over land below 60∘ latitude with a quality assurance value > 0.5, shortwave infrared (SWIR) albedo between 0.05 and 0.4 to avoid biases over dark scenes or highly reflective (often desert) scenes, a low blended albedo (< 0.75) to avoid snow-covered scenes (Wunch et al., 2011; Lorente et al., 2021a), and SWIR aerosol optical thickness less than 0.1. As shown in Fig. 3, retrieval count after filtering varies strongly by location. CHEEREIO regrids the native XCH4 observations on the fly to the GEOS-Chem grid resolution, as discussed later in this section.

A subset of the assimilation settings for this demonstration, passed to CHEEREIO through the configuration file, is listed in Table 2. The state vector includes weekly time-dependent global 3D concentrations as well as emissions on the 2∘ × 2.5∘ grid over land excluding poleward of 60∘. We use a 24-member ensemble, consistent with the LETKF ensemble size used for carbon fluxes in Liu et al. (2019). Each ensemble member is initialized with randomized methane emissions on the 2∘ × 2.5∘ grid that range from approximately 50 % to 150 % of prior values, based on a user-specified prior error parameter. Initial emissions for individual members are sampled from a normal distribution with spatial correlation and are normalized so that the initial ensemble mean emissions equal the prior emissions on the 2.0∘ × 2.5∘ grid. We spin up the model for each ensemble member for 4 months with these initial emissions and then further multiplicatively increase the ensemble standard deviation of methane concentrations by a factor of 5; the goal of this scaling is to emulate a much longer spin-up run of GEOS-Chem. We then adjust each ensemble member by the same global multiplicative factor so that the ensemble mean methane concentrations are equal to TROPOMI observations at the start of the assimilation period. Furthermore, we discard the first 2 months of assimilated output; we find that the LETKF system has a lag time between when assimilation begins (November 2018) and when emissions updates begin to stabilize, which we call the burn-in period. To reduce the time required for burn-in, for November and December 2018 we use a high regularization constant of γ=5 to artificially increase the weight of observations during the burn-in period.

The user does not specify how assimilation increments are split between emissions and concentrations: the LETKF formalism simultaneously updates different aspects of the state vector, emissions and concentrations included, solely according to the correlations between state vector elements represented in the prior error covariance matrix Pb, which is determined by the spread of the CTM ensemble. The prior error in concentrations is determined by the spread in concentrations resulting from the perturbed emissions in each ensemble member. Nevertheless, performance can be enhanced by establishing different parameters for different components of the state vector, such as using different localization radii or inflation schemes (Miyazaki et al., 2012b; Bisht et al., 2023).

Following ensemble spin-up and the burn-in months, we run the model with assimilation for 1 year (2019). We simultaneously assimilate 3D concentrations of methane as well as emissions; the LETKF algorithm natively computes prior error variance from the ensemble spread (for example, accounting for strong error correlation in the vertical). We use an assimilation period of 1 week and optimize grid cells following a horizontal localization radius of 500 km. We use an inflation factor Δ=0.03 and a regularization constant γ=1. Moreover, we impose a zero floor on emissions; a lognormal emissions spread (Sect. 3.2) would be a better way to prevent negative emissions (Maasakkers et al., 2019). We aggregate TROPOMI methane observations into super-observations on the GEOS-Chem 2.0∘ × 2.5∘ grid, reducing errors following Eq. (), with individual observation error σi = 17 ppb, transport error σtransport = 6.1 ppb, and error correlation c=0.28, which are values empirically determined for TROPOMI methane by Chen et. al. (2023).

Figure 4 shows the adjustment of global methane concentrations (Fig. 4a) and emissions (Fig. 4b) from the assimilation calculation relative to the prior emission inventory used in the control simulation. The control simulation produces methane concentrations slightly higher than observed by TROPOMI in January–June, leading to a downward correction of emissions. In July the situation reverses sharply as the control simulation falls well below TROPOMI observations, likely because of a seasonal underestimate of prior emissions from boreal wetlands and rice cultivation (Maasakkers et al., 2019). The assimilation responds with increased emissions but with a 1-month time lag reflecting the need to accumulate sufficient observations to inform the state vector. CHEEREIO's run-in-place capability (Sect. 3.3.1) would allow the LETKF algorithm to mitigate this lag, as would a sliding-window approach such as that used by the Carbon Tracker CH4 system (Bruhwiler et al., 2014; Liu et al., 2019).

Figure 5 shows the prior and posterior emissions for December 2019, along with the posterior error standard deviation. Figure 6 evaluates the ability of the posterior simulation to better fit the TROPOMI observations in that same month. Model bias is reduced by the LETKF assimilation procedure, with a mean bias of 1.2 ± 10.6 ppb in the ensemble (assimilated) mean compared to -16.1 ± 12.3 ppb in the prior (no assimilation) simulation.

The use of CHEEREIO to optimize methane emissions represents a substantial improvement in computational performance relative to an analytical inversion approach. Qu et al. (2021) previously applied the analytical approach with GEOS-Chem to optimize global methane emissions at 2∘ × 2.5∘ resolution for 2019. Their formulation had 4190 state vector elements, which required a total of 4190 perturbed GEOS-Chem simulations. By contrast, our approach only required 24 GEOS-Chem simulations to form the ensemble. Within CHEEREIO, the model spent an average of 29.8 % of wall time running GEOS-Chem and 70.2 % running LETKF routines. The relatively high overhead of LETKF routines occurs in part because the methane GEOS-Chem simulations are relatively fast; full oxidant–aerosol chemistry simulations are considerably more expensive but will have a similar LETKF overhead cost. Accounting for the relatively high LETKF overhead of CHEEREIO, we achieve a factor of 52× reduction in computational costs relative to an equivalent analytical inversion. On an annual basis, our results suggest a 6.5 % increase in global emissions relative to the prior, while Qu et al. (2021) suggest an increase of 2.6 % in global emissions for 2019, but they also decreased global tropospheric OH by 5.7 % (86 % of the total methane sink). Thus, our results are globally consistent. Some spatial patterns are consistent between the two inversions (increases in South and East Asia), while others are not (North America and Europe). Qu et al. (2021) used an older version of TROPOMI data more subject to retrieval artifacts, as documented by Lorente et al. (2021a). Further comparison of LETKF and analytical inversions using the same observations would be of interest.