Atmospheric inversion approaches are expected to play a critical role in future observation-based monitoring systems for surface fluxes of greenhouse gases (GHGs), pollutants and other trace gases. In the past decade, the research community has developed various inversion software, mainly using variational or ensemble Bayesian optimization methods, with various assumptions on uncertainty structures and prior information and with various atmospheric chemistry–transport models. Each of them can assimilate some or all of the available observation streams for its domain area of interest: flask samples, in situ measurements or satellite observations. Although referenced in peer-reviewed publications and usually accessible across the research community, most systems are not at the level of transparency, flexibility and accessibility needed to provide the scientific community and policy makers with a comprehensive and robust view of the uncertainties associated with the inverse estimation of GHG and reactive species fluxes. Furthermore, their development, usually carried out by individual research institutes, may in the future not keep pace with the increasing scientific needs and technical possibilities. We present here the Community Inversion Framework (CIF) to help rationalize development efforts and leverage the strengths of individual inversion systems into a comprehensive framework. The CIF is primarily a programming protocol to allow various inversion bricks to be exchanged among researchers. In practice, the ensemble of bricks makes a flexible, transparent and open-source Python-based tool to estimate the fluxes of various GHGs and reactive species both at the global and regional scales. It will allow for running different atmospheric transport models, different observation streams and different data assimilation approaches. This adaptability will allow for a comprehensive assessment of uncertainty in a fully consistent framework. We present here the main structure and functionalities of the system, and we demonstrate how it operates in a simple academic case.

The role of greenhouse gases (GHGs) in climate change has motivated an exceptional effort over the last couple of decades to densify the observations of GHGs around the world

In the past, research groups have developed various atmospheric inversion systems based on different techniques and atmospheric transport models, targeting specific trace gases or types of observations, as well as at various spatial and temporal scales, according to the particular scientific objectives of the study.
All these systems have their own strengths and weaknesses and help explore the range of systematic uncertainty in the surface to atmosphere fluxes.
Intercomparison exercises are regularly conducted to assess the strengths and weaknesses of various inversion systems

Despite their differences in methodology, application and implementation, almost all inversion systems rely on the same conceptual and practical bases: in particular, they use model–observation mismatches in a statistical optimization framework (most of the time based on Bayes' theorem) and numerical atmospheric tracer transport and chemistry models to simulate mixing ratios of GHGs and trace gases based on surface fluxes.
The objectives of the CIF are to develop a consistent input–model interface, to pool development efforts, and to have an inversion tool that is well documented, open-source and ready for implementation in an operational framework.
The CIF is designed to be a flexible and transparent tool to estimate the fluxes of different GHGs (e.g., carbon dioxide, CO

In the present paper, we lay out the basis of the CIF, giving details on its underlying principles and overall implementation.
The proof of concept focuses on the implementation of several inversion methods, illustrated with a test case.
We will dedicate a future paper to the evaluation of the system on a real-life problem with a number of interfaced atmospheric (chemistry–)transport models.
At the time of writing the present article, the following models are interfaced with the CIF: the global circulation models LMDZ

The version of the CIF presented here is implemented around Bayesian data assimilation methods with Gaussian assumptions, which constitute the main framework used in the atmospheric inversion systems for GHG fluxes and other trace gases

The Bayesian approach consists of estimating the following conditional probability density function (pdf):

As errors in inversion systems come from a large variety of independent causes superimposing on each other, it is often assumed that the most relevant way of representing the distributions in Eq. (

The assumption that errors are unbiased, which makes it possible to write normal distributions in Eq. (

The present version of the CIF includes three main categories of inversion methods: (1) analytical, i.e., algebraic solution of the unbiased Gaussian Bayesian problem; (2) ensemble methods with the ensemble square root filter (EnSRF); and (3) variational with two examples of minimizing algorithms (M1QN3 and CONGRAD).
Other types of data assimilation methods (e.g., direct sampling of probability density functions through Monte Carlo approaches) are also used by the community.
The choice of implementing the three aforementioned methods first is motivated by their dominant use and because these three use very different approaches for solving the Bayesian inversion problem, i.e., with/without random sampling of probability distributions and with/without the use of the adjoint of the observation operator.
The adjoint of the observation operator, noted

Analytical inversions compute the algebraic solution of the Gaussian Bayesian problem when it is linear, and they are used extensively at all scales

Analytical inversions can also be used on slightly non-linear problems, by linearizing the observation operator around a given reference point using the tangent linear of the observation operator.
It formulates as follows:

Then Eq. (

The computation of an analytical inversion faces two main computational limitations.
First, the matrix

Depending on the number of available observations or the size of the target vector, one of the two is preferred to limit the number of observation operator computations to be carried out explicitly.
When the dimension of the target vector is relatively small, the response function is generally preferred; conversely, when the observation vector is small, the footprint approach is preferred.
The type of transport model used to compute the matrix

The second obstacle consists of the fact that the computation of the Kalman gain matrix in Eq. (

Ensemble methods are commonly used to tackle high-dimensional problems and to approximately characterize the optimal solution.
In ensemble methods, such as ensemble Kalman filters (EnKFs) or smoothers

Observations are first assimilated sequentially in the system to reduce the dimension of the observation space, making it possible to explicitly compute matrix products and inverses, and thus propagating information from the target space to the observation space;
the overall inversion period is processed incrementally using a smaller running assimilation window including a manageable number of observations; intermediate inversions are solved on the smaller running window that is gradually moved from the beginning to the end of the overall data assimilation window;
the running assimilation window with so-called analysis and forecast steps introduces complex technical challenges to rigorously propagate errors from one iteration of the running window to the next one;
moreover, the sequential assimilation of observations is valid only under the assumption that observation errors are not correlated between assimilation windows, which may prove incorrect for high-density data sets, but is an assumption also done in, for example, variational inversions.
For very dense observations, such as datasets from new-generation high-resolution satellites, the sequential assimilation of observations may not be sufficient, or at least methods may be needed to make the observation errors between sequential assimilation windows independent, e.g., by applying a whitening transformation to the observations to form a new set with uncorrelated errors as suggested by

The posterior distribution at a given step of the filter is then characterized explicitly by applying Eq. (

In the atmospheric inversion community, another ensemble method is widely used, based on the CarbonTracker system

From there, Eq. (

By using random sampling, ensemble methods are able to approximate large dimensional matrices at a reduced cost without using the adjoint of the observation operator (see variational inversion below) that can be challenging to implement.
Small ensembles generally cause the posterior ensemble to collapse; i.e., the posterior distribution is dominated by one or a very small number of members, which does not allow for a reliable assessment of the posterior uncertainties

In the current version, only the EnSRF approach is implemented in the CIF.
One should note that the EnSRF, as a direct approximation of the analytical solution, can be very sensitive to non-linearity in the observation operator

Variational inversions use the fact that finding the mode of the posterior Gaussian distribution

In variational inversions, the minimum of the cost function in Eq. (

Quasi-Newtonian methods are a group of algorithms designed to compute the minimum of a function iteratively.
It should be noted that in high-dimensional problems it can take a very large number of iterations to reach the minimum of the cost function

Solving Eqs. (

Overall, variational inversions are a numerical approximation to the solution of the inversion problem: they involve the gradient of the cost function in Eq. (

The variational formulation does not require calculation of complex matrix products and inversions, contrary to the analytical inversion, and is thus not limited by vector dimensions.
Still, the inverses of the uncertainty matrices

When the observation operator is linear, the posterior uncertainty matrix

Another approach to quantify the posterior uncertainty matrix

Forward simulations simply use the observation operator to compute simulated observation equivalents.
It reads as

This mode is used to make quick comparisons between observations and simulations to check for inconsistencies before running a full inversion. It is also used by the analytical inversion mode to build response functions.

The test of the adjoint is a crucial diagnostic for any inversion system making use of the adjoint of the observation operator.
Such a test is typically required after making any edits to the code (to the forward observation operator or its adjoint) before running an inversion.
Coding an adjoint is prone to errors and even small errors can have significant impacts on the computation of the gradient of the cost function in Eq. (

In practice, the two terms of the equation are rarely exactly equal.
Nevertheless, the difference should never exceed a few times the machine epsilon. Besides, Eq. (

Elementary operations required for each data assimilation method. An: analytical inversion; EnKF: ensemble Kalman filter; Var: variational; Fwd: forward simulation; AdTest: test of the adjoint. We note

Each inversion algorithm and computation mode mentioned above can be decomposed into a pipeline of elementary transformations.
These transformations are listed in Table

Call structure of the CIF.

Avoiding redundancy makes the maintenance of the code much easier and provides a clear framework for extensions to other inversion methods or features. For instance, inverse methods based on probability density functions other than normal distributions could be easily implemented by updating the random ensemble generator or by implementing new cost functions representing non-Gaussian distributions while keeping the remaining code unmodified. In particular, non-Gaussian cost functions still rely on the computation of the observation operator. New combinations of elementary transformations can also directly lead to new methods.
For instance, ensemble variational inversion

The complexity of the selected elementary transformations spans a wide range, from one-line straightforward code to computationally expensive and complex code implementation. In small dimensional and/or linear problems, the computation of the observation operator using its Jacobian and matrix products may be computationally expensive, but it is in principle rather straightforward to implement. For non-linear and/or high-dimensional problems, these transformations require simplifications and numerous intermediate steps. For instance, applying matrix products to the error covariance matrices

In any case, the observation operator (see details in Sect.

The observation operator is a key component of all inversion methods.
It links the target space to the observation space, and conversely, its adjoint links the observation space to the target space.
To do so, the observation operator projects its inputs through various intermediate spaces to the outputs. As atmospheric inversions need a representation of the atmospheric transport (and chemistry if relevant) to link the target vector (including surface fluxes, atmospheric sources and sinks, initial and boundary conditions for limited domains and time windows, etc.) to the observation vector (including some form of atmospheric concentration measurements), the observation operator is built around a given CTM in most cases: Eq. (

The targeted structure of the CIF should allow for a full flexibility of observation operators, from the straightforward widely used decomposition detailed in Eq. (

In such a formalism, all intermediate transformations have the same conceptual level in the code. They are functions ranging from spatial reprojection to temporal interpolations and to more complex operations such as the reconstruction of satellite total columns from concentrations simulated at individual levels in the transport model. In the CIF, all these transformations have the same input and output structure; thus, their order can be changed seamlessly to execute a given configuration. Please note that the commutative property of elementary transformations as pieces of code does not guarantee the commutative property of the corresponding physical operators.

Such a transformation-based design allows us to rigorously separate transformations and thus to implement and test their respective adjoints more easily.
Once adjoints for each individual operation are implemented, the construction of the general adjoint is straightforward by reversing the order of forward operations:

Figure

Observation operator structure. Emissions are processed from the target vector to generate model inputs, as well as other inputs, not optimized by the inversion; in this example, some background for the simulations is also optimized by the inversion and is added to simulations at the end of the pipeline just before stacking outputs to the observation vector format.

The mathematical formalism of Eqs. (

Organization of the Community Inversion Framework.

The Community Inversion Framework project follows the organization scheme of Fig.

The

Test cases (including the ones presented in Sect.

To reflect the theoretical flexibility required in the computation of various inversion methods and observation operators, we made the choice of implementing

General classes of objects emerge from the definition of the abstract structure of the inversion framework.
These classes are defined by the data and metadata they carry, as well as by the methods they include and their interaction with other classes.
The main classes are the following:

Details on metadata and operations for each class are given in the Supplement, Table S1. Our objective was to design a code that is fully recursive in the sense that modifying some instance of a class does not require users to update other classes calling or being called by the modified class. Thus, each class is built so that it only needs internal data, as well as data from the execution level just before and after it, in order to avoid complex dependencies while allowing proper recursive behaviour in building the transformation pipeline. For instance, the observation operator applies a pipeline of transformations from the target vector to the observation vector. Some transformations will use the model class to run the model or the domain class to carry out reprojections, or the target vector to aggregate/de-aggregate target dimensions, etc. Despite the many complex transformations carried out under the umbrella of the observation operator, only the sub-transformations of the pipeline are accessible at the observation operator level, which do not have to directly carry information about, for example, the model or other classes required at sub-levels. This makes the practical code of the observation operator much simpler and as easy to read as possible.

To translate the principle scheme of Fig.

For each plugin required in the configuration (primarily the computation mode),

For instance, in the call graph in Fig.

In practice,

To set up a

In the following we describe a demonstration case based on a simple implementation of a Gaussian plume dispersion model and simple inversion set-ups. The purpose of this demonstration case is a proof of concept of the CIF, with various inversion methods. We comment and compare inversion set-ups and methods for the purpose of the exercise, but conclusions are not made to be generalized to any inversion case study due to the simplicity of our example. The test application with a simple Gaussian plume model allows users to quickly carry out the test cases themselves, even on desktop computers, to familiarize themselves with the system. Nevertheless, the Gaussian plume model is not only relevant for teaching purposes but also for real applications, as it is used in many inversion studies from the scale of industrial sites with in situ fixed or mobile measurements

Gaussian plume models approximate real turbulent transport by a stable average Gaussian state

To illustrate atmospheric inversions, we use a grid of point surface fluxes to simulate concentrations using the Gaussian plume equation.
Thus, the concentration at a given point and time

This formulation highlights the linear relationship between concentrations and fluxes.
As the concentrations can be expressed as a matrix product, the computation of the adjoint of the Gaussian model is straightforward and does not require extra developments:

For the purpose of our demonstration cases, meteorological conditions (wind speed, wind direction and stability class) are randomly generated for the simulation time window. Fixed seeds are selected for the generation of random conditions in order to make them reproducible.

Posterior increments for analytical, EnSRF, variational with CONGRAD and variational with M1QN3 (from top to bottom, left to right) for an inversion set-up at the pixel resolution with horizontal correlation length of 500 m.

Cases discussed in Sect.

Truth observations are generated by running the Gaussian model in forward mode with known fluxes defined as the sum of prior fluxes

A random noise of 1 % of the standard deviation of the forward simulations was added to the truth observations to generate measurements. Please note that the perturbation of the fluxes is generated using an explicit formula and not a random perturbation with a given covariance matrix. We discuss results with different possible target vectors and covariance matrices.

Uncertainty reduction for analytical, EnSRF, variational with CONGRAD and variational with M1QN3 (from top to bottom, left to right) with the same set-up as in Fig.

The objective of our test case is to prove that our system enables users to easily compare the behaviour of different inversion methods in various configurations. To do so, we conduct three sets of four inversions for the demonstration of our system.
Each set includes one analytical inversion, one EnSRF-based inversion and two variational inversions based on M1QN3 and CONGRAD minimization algorithms respectively.
The sequential aspect of the EnSRF is not implemented in the CIF; hence, the comparison with the other inversion methods only includes the random sampling of the target vector distribution to solve Eq. (

The three sets of inversions differ by the dimension of the target vector and the spatial correlations of errors.
The first set uses a target vector based on a grid of

To assess the sensitivity of each set-up to the allocated computational resources, we computed the EnSRF and the two variational inversions with varying numbers of simulations

In the following, we present detailed figures for the test case at the pixel resolution with a correlation length of 500 m. For the sake of readability, figures for other test cases are grouped in Sect. S2 of the Supplement.

Posterior increments are presented in Fig.

Prior

In the case with the target vector aggregated on groups of pixels, all inversion methods converge towards a very similar solution.
In this case, the posterior increments reproduce the overall structure of the truth–prior difference, with one local minimum in the centre of the domain.
However, the aggregated target vector results in too coarse patterns which are not representative of the actual truth–prior difference.
In the case with the target vector at the grid's resolution with spatial correlations of 500 m, all methods capture well the true-prior difference structure.
However, posterior increments are rather noisy compared to the truth.
This is due to the spatial correlations being inconsistent with the smooth perturbation with fixed length scales in Eq. (

In all cases, CONGRAD converges at a faster pace than the other two methods, and after a limited number of iterations, the convergence rate is close to zero, suggesting additional simulations do not provide significant additional information to CONGRAD (although additional iterations bring new constraints on the posterior uncertainty matrix).

Overall, CONGRAD appears to be the most cost-efficient algorithm to estimate the analytical solution in the case of a linear inversion in our very simple demonstration case. Though not as efficient, the EnSRF method can approximate the analytical solution at a reduced cost. By design, its computation can easily be parallelized, which can allow for a faster computation than CONGRAD when computational resources are available in parallel. M1QN3 proves not as efficient as its CONGRAD counterpart, but contrary to CONGRAD, it can accommodate non-linear cases.

The reduction of uncertainties and posterior uncertainty matrices are shown in Figs.

Cost function evaluation for varying numbers of computed simulations for analytical (red), EnSRF (green), variational with CONGRAD (orange) and variational with M1QN3 (blue) methods.

We have introduced here a new generic inversion framework that aims at merging existing inversion systems together, in order to share development and maintenance efforts and to foster collaboration on inversion studies. It has been implemented in a way that is fully independent from the inversion configuration: from the application scales, from the species of interest, from the CTM used, from the assumptions for data assimilation, and from the practical operations and transformations applied to the data in pre- and post-processing stages. This framework will prevent redundant developments from participating research groups and will allow for a very diverse range of applications within the same system. New developments will be made in an efficient manner with limited risks of unexpected side effects, and thanks to the generic structure of the code, specific developments for a given application can be directly applied to other applications. For instance, new inversion methods implemented in the CIF can be directly tested with various transport models. With modern inversion methods moving towards a hybrid paradigm of variational and ensemble methods, the flexibility of the CIF will be a valuable asset as abstract methods can be easily mixed with each other.

We have presented the first version of this Community Inversion Framework (CIF) alongside its Python-dedicated library

The next step of the CIF is the implementation of a large variety of CTMs. The implementation of new CTMs already interfaced with other inversion systems should not bring particular conceptual challenges as all interface operations are already written in their original inversion system; in most cases, re-arranging existing routines is sufficient to interface a model to the CIF. One particular challenge concerns I/O optimizations: the generation of inputs and the processing of outputs can be time consuming and in some very heavy applications require specific numerical and coding optimizations. The very general formalism of the CIF may hamper the ability of applying these particular optimizations for some models. Best efforts will have to be deployed to take full advantage of advanced I/O and data manipulation libraries in Python to limit this weakness.

CHIMERE, LMDZ, TM5, FLEXPART, and STILT have already been implemented, and a sequel paper will evaluate and compare their behaviour in similar inversion set-ups. COSMO-GHG and WRF-Chem are also planned to be implemented in the near future to widen the developer and/or user community of the system. The use of various CTMs in identical inversion configurations (inversion method, observation and target vector, consistent interface, etc.) will allow for extensive assessments of transport errors in inversions. Despite many past efforts put into inter-comparison exercises, such a level of inter-comparability has never been reached and will be a natural by-product of the CIF in the future. In addition, comparing posterior uncertainties from different inversion methods and set-ups will make it possible to fully assess the consistency of different inversion results.

The flexibility of the CIF allows very complex operations to be included easily. They include the use of satellite observations, which will be evaluated in a future paper, inversions using observations of isotopic ratios and optimizing both surface fluxes and source signatures

The code files, documentation pages (including installation instructions and tutorials) and demonstration data used in the present paper are registered under the following link

The supplement related to this article is available online at:

All authors contributed to the elaboration of the concept of Community Inversion Framework. AB designed the structure of the CIF.
AB, ES, IP and JT are the main developers of the Python library

The authors declare that they have no conflict of interest.

Publisher’s note: Copernicus Publications remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The authors thank the reviewers (Peter Rayner and an anonymous referee) for their fruitful comments on our manuscript.

The Community Inversion Framework is currently funded by the project (

This paper was edited by Christoph Knote and reviewed by Peter Rayner and one anonymous referee.