The CarbonTracker data assimilation system for CO2 estimates carbon fluxes between the atmosphere and the surface (land biosphere and oceans), using observations of atmospheric CO2 mole fractions. At its core, CarbonTracker is an ensemble Kalman smoother application using a fixed-lag assimilation window of which several flavors are used in trace gas studies e.g.. The surface CO2 fluxes are optimized using the cost function (J) that describes the system according to J(x)=(yo-H(x))TR-1(yo-H(x))+(x-xb)TP-1(x-xb), where y are the atmospheric CO2 mole fraction observations, with their error covariance R. H is the observation operator (TM5) that connects the observations yo to the scalars that modify the surface CO2 fluxes, which are contained in the state vector x. Prior information on the surface fluxes is contained in the background state vector xb with error covariance P. Ensemble statistics are created from 150 ensemble members, each with its own background CO2 mole fraction field. The length of the smoother window (“lag”) is set to 5 weeks. Flux patterns within regions with good observational coverage (e.g. Europe and North America) are robustly resolved well within that time, while regions with low observational coverage are less well constrained. We refer the reader to previous publications and the web page (http://www.carbontracker.eu/documentation.html) for further general details on the ensemble Kalman smoother as applied in CarbonTracker.

CarbonTracker started with CO2 data assimilation included in the TM5 Fortran code. With ongoing developments in CarbonTracker, we required a more flexible data assimilation framework, that could accommodate more complex data flows and structures, and be applied to other applications. Such frameworks for data assimilation exist, and have been successfully used across a range of applications. One example of a popular data assimilation package is the Data Assimilation and Research Test bed, DART see http://www.image.ucar.edu/DAReS/DART;. It offers many out-of-the-box options for data assimilation and supports a wide range of platforms and possible applications. These are primarily, but certainly not limited to, meteorological data assimilation efforts and include ensemble systems oriented on atmospheric constituents e.g.. Another example is the openDA toolkit resulting from initial developments at Delft University (http://www.openda.org/joomla/index.php), which initial focus was on hydrological applications, but was expanded to also include wave models and air quality models. Furthermore, the European Centre for Medium-Range Weather Forecasting (ECMWF) is currently developing the Object-Oriented Prediction System (OOPS) framework , which is used in their Integrated Forecasting System (IFS). These open-source frameworks aim to provide their users with an easy-to-use and well-documented data assimilation system, and in that sense would be suitable for CarbonTracker as well. However, the CarbonTracker system is characterized by a long-lag window of several weeks, and by a very expensive observation operator (i.e. a TM5 simulation). Since the application of an ensemble Kalman smoother is also not provided by any existing open-source system, we decided to implement our own data assimilation shell.

Looking at the requirements for our CTDAS, we realized that the Python language could handle the tasks needed such as basic shell scripting, use of numerical recipes, job control under UNIX, I/O in NetCDF and HDF, analysis and visualization, and even remote interfacing over TCP/IP and HTTP. Pythons' functionality for object-oriented implementation moreover suited well our desired modular design of CTDAS, with minimal code duplication and efficient use of class inheritance to build diverse pipelines for data assimilation. Specifically, we aimed to make CTDAS:

independent of application (carbon dioxide, methane, isotope ratios, or multi-tracer);

The CTDAS code is based on the use of seven Python classes (or templates to create objects), each representing a different part of the data assimilation system. They are visualized in Fig. . Three classes are referred to as “control” classes, as the objects they instantiate are used to control the ensemble data assimilation system. These three control classes are

Class CycleControl controls the cycling through time, succession of cycles and organization of input and output data, including checkpointing data, for each cycle. This is the only core object of CTDAS that is automatically created based on options and arguments passed along when submitting the main CTDAS job (e.g. cycle length, smoother window length (lag) and number of ensemble members).

The specific details for a given experiment are controlled through external run-control files (rc-files), which consist of key:value pairs that pass information to CTDAS on, e.g., the dates for which to run the experiment or the number of parameters (scaling factors) and ensemble members. For each of the three control classes CTDAS provides a “base class” describing the required methods, attributes and the expected interface when accessing these from within CTDAS. Specific applications can then inherit these base classes, and modify only those methods or attributes that differ for their specific configuration. For example, a Platform object with a method to submit a job script with a proper command (e.g. sbatch) to a specified queueing system (e.g. SLURM) can be used for a high-performance computing environment. This same method in the Platform object could similarly prepare a job script for the next cycle on a regular workstation, but in that case could, e.g., simply spawn a new task (sh) for this job.

The four classes that complete CTDAS are

Class StateVector builds the data structure of a state vector, defined by three dimensions in parameter space (number of scaling factors, ensemble members and lag), including sampling of random ensemble members from a specified distribution.

The class Optimizer currently supports two versions of the square root ensemble Kalman smoother originally presented in Whitaker and Hamill (2002) and Peters et al. (2005), both for an observation serial algorithm and a batch algorithm. In the latter, the Kalman filter equations are solved using matrix expressions of K (the Kalman gain matrix), R and HPHT rather than scalar or vector values. This can be useful when observation errors are correlated (a non-diagonal matrix R). Other optimization methods (e.g. 4-D variational approach) have so far not been implemented in CTDAS, but can be added with relatively little effort by creating a new Optimizer class.

Special attention is focused on the ObservationOperator, which consumes the majority of CPU in CarbonTracker, and was previously TM5 by definition because it was the heart of the code base. Here, we have explicitly made the observation operator external to the CTDAS code and call it from a separate class. This allows TM5 to be replaced by a different transport model in CTDAS, and also enables development and maintenance of the TM5 code separate from CTDAS. In the currently implemented TM5 ObservationOperator class, an external call compiles the TM5 transport model (using Fortran and a set of TM5-specific control scripts), and this precompiled TM5 executable is subsequently called to simulate mole fraction needed in the ensemble Kalman smoother. Control of TM5 is taken over by the CycleControl object, which modifies TM5-specific input data for the current data assimilation cycle (e.g. begin and end time). The Platform object allows TM5 jobs to be run in parallel operation through the queuing system, and once finished returns control to the main Python program (CTDAS itself is currently not parallelized). This job flow is further explained in the next section, but we stress here that all references to TM5 in this paragraph can easily be replaced by that of any other transport model (e.g. WRF, GEOS-Chem or even Lagrangian transport models like STILT) as long as there is an appropriate ObservationOperator class.

The seven classes described above are imported as objects in the main Python script, which subsequently calls a “pipeline” script with these objects as arguments. The pipeline takes care of the order in which all steps of an experiment are performed. A key property of the pipeline is that all calls to methods in external modules (i.e., function calls) are generic, rather than specific. This means for instance that to achieve a simulation of the transport model, the generic method (e.g. run_simulation()) of an ObservationOperator is called rather than an application-specific method (such as run_tm5_with_co2()). The pipeline will therefore work for any ObservationOperator class with a properly programmed interface, and can be independent of specific implementations of a transport model.

The objects used in CTDAS can not only be tailored to a specific application, but also be combined in different ways, yielding different pipelines. An example is the simple “forward” pipeline, which combines the complementary Observations, StateVector and ObservationOperator objects with the three control classes. The forward pipeline simulates forward transport (ObservationOperator) of a given tracer as controlled by specified inputs (such as emission scaling factors) in the StateVector, while sampling mole fractions at all times and locations included in the Observations object. This sequence is repeated for all time steps specified in CycleControl, until the final cycle is reached. Another example is the “analysis” pipeline, combining Observations and StateVector objects with the three control classes, to extract the results from an experiment to convenient output formats (e.g. aggregated fluxes for defined regions).

A more complex pipeline, important to this paper, is the inverse pipeline that yields an actual optimization result. The pseudo-code that achieves this in CTDAS (similar to the illustration in Peters et al., 2005) is

Create the seven objects from the code structure (note that the first is automatically created from options and arguments when submitting the main CTDAS job; see Sect. ):DaCycle = CycleControl(opts, args)DaSystem = da.carbondioxide.dasystem()PlatForm = da.platform.cartesius()Observations = da.observations.obspack_obs()StateVector = da.co2gridded.statevector()ObsOperator = da.tm5.observationoperator()Optimizer = da.baseclasses.optimizer()

The CTDAS system is documented using the open-source SPHINX package (http://sphinx-doc.org) that can export documentation written inside the code itself to various output formats including HTML, PDF, RTF and more. The output of CTDAS documentation can be viewed at http://www.carbontracker.eu/ctdas/. An important advantage of this inline documentation is that the code and its description exist within the same text files, and are thus more easily updated together. This is preferably done at the same time that the source code is modified, by the programmer doing the actual modifications. Because the syntax of this documentation is relatively simple (SPHINX handles the translation to nicely readable document formats), the burden on code developers is minimal.

CarbonTracker estimates weekly scaling factors (λr) for both net biome exchange (NBE) and net ocean exchange, using atmospheric observations of CO2 mole fractions from a global observing network. The total carbon fluxes F(x,y,t) for each region r (defined by longitude x and latitude y) and each time step (t) are represented by F(x,y,t)=λr⋅Fbio(x,y,t)+λr⋅Foce(x,y,t)+Ffossil(x,y,t)+Ffire(x,y,t). The scaling vectors (λr) multiply Fbio and Foce, which are pre-calculated space–time patterns obtained from biosphere and ocean models (prior fluxes). Fossil fuel (Ffossil) and biomass burning (Ffire) emissions are not scaled/optimized. The monthly mean prior ocean fluxes in CTE2016 are from the ocean inversion by . Earlier versions of CarbonTracker used prior biosphere and fire carbon fluxes from the CASA-GFED2 system . In CTE2016 this has been replaced by the SiBCASA-GFED4 model . SiBCASA-GFED4 provides net carbon fluxes for the dominant vegetation type in each 1∘ × 1∘ grid box globally for every 3 h. Daily fire emissions are included in these biosphere model calculations based on satellite observed burned area . The seasonal development of vegetation is scaled with the satellite observed greenness (normalized difference vegetation index, NDVI) and absorption of radiation (fPAR). The fossil fuel emissions are from the , together with worldwide country- and sector-specific time profiles derived by the Institute for Energy Economics and the Rational Use of Energy (IER) from the University of Stuttgart and constructed for the CARBONES project (http://www.carbones.eu/). The global total fossil fuel emissions are scaled with different regional annual trends for each continent to global totals as used in the global carbon budget of the Global Carbon Project (GCP).

These prior fluxes are transported with the TM5 transport model on a global resolution of 3∘ × 2∘ with zoom regions of 1∘ × 1∘ over Europe and North America. TM5 uses meteorological driver data from the ERA-Interim reanalysis from the ECMWF. The convective entrainment and detrainment fluxes are obtained directly from the ERA-Interim data, whereas in earlier versions we used the Tiedtke convection scheme . The resulting CO2 mole fractions are compared to atmospheric CO2 observations and their differences are minimized using the ensemble Kalman smoother (using 150 ensemble members), by adjusting the flux scaling vectors (λr) resulting in optimized posterior fluxes. The background scaling factors (λb) for each new time step t are chosen as the average of the optimized scaling factors (λa) from the two previous time steps, and the fixed prior value, as in λtb=λt-2a+λt-1a+λp/3.0. The CO2 mole fraction observations are from the ObsPack product: GLOBALVIEWplus v2.1 . CTE2016 assimilates discrete (flask) samples as well as hourly values for well-mixed conditions (afternoon hours for most locations, and nighttime hours for mountain locations).

The current setup of CarbonTracker Europe for CO2 (CTE2016) has several differences compared to the current version of CarbonTracker at NOAA (CT2016). We document here the most important differences:

CTE2016 uses CTDAS, CT2016 uses the implementation in TM5.

Previous releases of CarbonTracker applied the same scaling factor for the biosphere fluxes (λr) to all grid boxes that share the same “ecoregion” type, which means they have a similar dominant land-cover type within a broader continental region (e.g. European Croplands). The land-cover types are defined by the Olson ecosystem classification , and the continental regions follow the TransCom definitions . This approach implies that errors in the pre-calculated biospheric fluxes are fully correlated over the ecoregion, and adjustments needed to match atmospheric CO2 mole fractions must be applied to all grid boxes of that ecoregion (proportional to the magnitude of the flux because of the linear scaling). Although this might be realistic within the context of the biosphere model that uses the same parameterizations for the same land-use types, this assumption can be questioned for actual carbon fluxes. Especially when ecosystems are geographically far apart (such as coniferous forests along the east and west coast of boreal North America), their responses to similar weather forcings might be quite different because of differences in, e.g., age structure, or management regime.

A more realistic alternative is to assume no error correlations in the biosphere fluxes over space, an approach supported by independent research based on observations . However, since the density of the observing network does not allow each ecosystem in the world to be monitored and optimized independently, many other data assimilation systems assume that correlations between regions decay exponentially as a function of distance. This correlation length scale is chosen mostly based on practical considerations, and can vary from a few 100 km to more than 1000 km e.g.. Effectively, this correlation strongly reduces the number of degrees of freedom in the covariance matrix (Pb) of the scaling factors, balancing it with the number of observations. For instance, a gridded state vector for land fluxes at 5∘ × 4∘ resolution has around 1000 land grid boxes, but only about 60 degrees of freedom when using a length scale of 1000 km .

In CTDAS, we adopted this approach, enhanced with ecoregion information through the covariance, and implemented a gridded state vector for the Northern Hemisphere land regions on 1∘ × 1∘ resolution. We still apply the region-based state vector to all ocean regions as well as the Southern Hemisphere ecoregions. To manage the degrees of freedom we use this approach only for the land TransCom regions of the Northern Hemisphere which are best constrained by observations, and we furthermore use variable length scales reflecting this observation network density. Moreover, in TransCom regions with a gridded state vector we limit the correlations to exist only between grid boxes within the same Olson ecoregion , such that a priori errors in forest fluxes do not correlate with errors in crop fluxes even if they are dominant in neighboring grid boxes. The chosen prior standard deviation (σ, SD) is 80 % on land parameters, and 40 % on ocean parameters, reflecting more prior confidence in the ocean fluxes than in terrestrial fluxes, because of the lower variability and larger homogeneity of the ocean fluxes. The maximum covariance is therefore 0.64 (σ2) for land parameters. The structure of the new gridded state vector is summarized in Table , showing a total number of 9835 scaling factors to be estimated each week, with close to 1100 degrees of freedom. An example of the covariance for a specific grid box in the European conifer forest region is given in Fig. .

Within the new CTDAS system, the implementation of this new gridded state vector required the creation of (1) a new global map that numbers each 1∘ × 1∘ grid box according to its associated state vector element (N=1,…,9835), and (2) an a priori covariance matrix for this new state vector, (3) a new DaSystem class (see Sect. ) that defined the state vector size for this new configuration and, finally, (4) a new StateVector class (GriddedStateVector), which inherited all methods from the base class StateVector (see Sect. ), and in addition had modified methods to efficiently read the covariances and create ensemble members. This implementation is also flexible and can be used easily in other applications with different setups of the state vector (see Sect. ).

We have started providing annual releases of the carbon flux estimates from CarbonTracker Europe since 2013. The current version is CTE2016 and includes carbon flux estimates for 2001–2015. CTE2016 uses the gridded state vector (Sect. ). Other general details of the setup and e.g. prior fluxes are described in Sect. . Carbon fluxes are estimated for the period 2001–2015 and are shown annually for the global scale in Fig. . This figure shows the imposed fossil fuel and biomass burning emissions and the resulting net ocean and land sinks. The natural CO2 sinks show considerable interannual variability, mainly due to climatic differences between the years. Since the land and ocean sinks are calculated from the emissions and the observed atmospheric CO2 mole fractions, they reflect the interannual variability in the atmospheric growth rate. Figure  also shows the comparison of the total fluxes estimated by CTE2016 with the global atmospheric CO2 growth rate as observed at background sites from the NOAA ESRL network . The growth rates are converted from ppmyr-1 to PgCyr-1 using 2.12 PgCppm-1 . The total fluxes from CTE2016 match the observed atmospheric growth rate and its interannual variability well (up to 0.3 ppm yr-1). The remaining differences reflect differences not only due to observation sites included in either the data assimilation or the calculation of the global growth rate, but also due to, e.g., transport model errors and a time delay, since fluxes of the end of a year influence the atmospheric growth rate of the next year.

The fossil fuel emissions increased from 6.8 PgCyr-1 in 2001 to 9.8 PgCyr-1 in 2015. The fossil fuel emissions in 2014 and 2015 are almost equal, but the 2015 atmospheric growth rate of 2.98±0.09 ppm yr-1 is much higher, compared to 1.99±0.09 ppm yr-1 in 2014. As shown in Fig. , CTE2016 assigns this anomaly to a smaller net uptake by the biosphere, and in a lesser extent to a smaller net ocean uptake. Biomass burning emissions have also slightly increased between 2014 and 2015.

Over the period 2001–2015, especially 2011 and 2014 stand out with high net land uptake, and the net carbon sinks in 2002, 2003 and 2005 were relatively low (Fig. ). Figure  shows the annual development of the cumulative anomalies of the net natural carbon fluxes (biosphere and ocean sinks, and the emissions from biomass burning). These anomalies are the deviations from the 2001–2015 mean. In 2011 and 2014, the sinks were relatively larger throughout the year. The year 2015 had higher than average net uptake in summer, but this effect was canceled by a reduced net uptake in the remainder of the year.

Both natural sinks show an increasing trend over the period 2001–2015. The average net ocean sink slightly increased from -1.9±0.8 PgCyr-1 in 2001–2003 to -2.5±0.1 PgCyr-1 in 2013–2015. The average net land sink (including biomass burning emissions) increased from -1.8±1.0 to -2.3±0.8 PgCyr-1 over the same time periods. Global maps of the ocean and biosphere fluxes (including biomass burning emissions) for the prior and posterior estimates averaged over the 2001–2015 period are shown in Fig. . The average posterior net biosphere sink (excluding biomass burning emissions) over 2001–2015 of -3.8 PgCyr-1 is larger compared to the prior estimate of -2.4 PgCyr-1. The sink especially increases in the Northern Hemisphere. The average net ocean sink of -2.3 PgCyr-1 is lower than the prior estimate of -2.7 PgCyr-1, and also the trend in the ocean sink decreases from a prior estimate of -0.075 to -0.044 PgC yr-2.

Figure  shows the latitudinal distribution of the average and SD of the residuals of the simulated minus observed CO2 mole fractions for all assimilated observations. With the exception of a few sites, the remaining biases are generally small and well below 1 ppm. The SD is largest in the Northern Hemisphere mid latitudes. The mean bias over all sites is 0.027±0.67 ppm, and the average of the absolute values of the biases is 0.31±0.59 ppm. There is a difference in the bias between the summer and winter, as the wintertime observations are generally better represented in CarbonTracker because of the lower variability in the CO2 concentrations in winter, together with lower transport errors because of the difficulties in representing the smaller-scale convective transport during summer. CTE2016 overestimates the CO2 mole fractions in the Northern Hemisphere summer and the average bias is 0.31±0.89 ppm. In the Northern Hemisphere winter this is -0.13±0.65 ppm.

Although CTE2016 optimizes fluxes on the global scale, carbon fluxes can also be estimated for smaller (eco)regions. Figure  shows the net carbon sink of the European forest ecoregion over the period 2001–2015, together with the emissions from fossil fuels from the same region. Forest areas and human activities strongly overlap in Europe (on 1∘ × 1∘ resolution). In most of the years the forests take up carbon from the atmosphere and thereby partly compensate the emissions. The average European carbon sink over 2001–2015 is 0.17±0.11 PgCyr-1, with some interannual variability and especially in years with droughts, like 2003 or 2010, the net European forest carbon sink is reduced to zero. Other (eco)regions in Europe (specifically grasslands) are close to neutral, while croplands can add up to a small source in some years. Our forest carbon sink is in good agreement with , but not with the space-based estimate from , who find a larger sink in European forests.

The first release of carbon flux estimates from CTE was in 2008 (CTE2008). Table  gives an overview of the different versions of CTE. Generally, the version IDs include the year in which the version is released and the simulation covers the years from 2001 until the year before the release date (e.g. CTE2008 covers 2001–2007, while CTE2013-OD is an exception and covers 2001–2010). Simulations start in 2000, which is discarded and seen as a spin-up of the calculations. CTDAS (Sect. ) was used for all versions from CTE2013-OD. Since 2014, CTE results have been included in the annual updates of the global carbon budget published by the GCP CTE2014, CTE2015, CTE2016-FT in resp. .

From version CTE2008 to version CTE2016, several changes have been implemented. Most of the prior fluxes have been changed, except for the ocean prior fluxes, and the amount of observations and observational sites has increased. The most significant updates are (1) the implementation of the gridded state vector from version CTE2013 (Sect. ), and (2) changes in the TM5 meteorology, including (a) changing from operational data from the ECMWF to using ERA-Interim reanalysis driver data , and (b) the use of the convective entrainment and detrainment fluxes directly from ECMWF from version CTE2014, instead of using the previous Tiedtke convection scheme.

The differences in the estimated natural carbon fluxes (ocean and biosphere including biomass burning emissions) between the different versions are shown in the left panel of Fig.  for selected regions for their overlapping period 2001–2007. The posterior uncertainty in CarbonTracker can be estimated by different approaches. The right panel includes the fluxes for a single region (northern land) together with two options for the uncertainty estimate. The first option shows the internal error based on the weekly posterior covariance matrix. A new prior covariance is included for each new week in the inversion, not taking into account information on the uncertainty (reduction) in the previous weeks. This results in a unrealistically large error estimate due to the absence of temporal correlation of the covariance in combination with the short assimilation window. The advantage is that fluxes from different regions remain uncoupled in new weeks. Alternatively, the uncertainty of an inversion can be estimated by the range between estimates from several different realizations e.g.. The second option in Fig.  shows the range between the seven versions of CarbonTracker Europe. This is our preferred option and is also used in and . The resulting carbon fluxes from these versions show differences based on the choices made in their setups. In the most recent version CTE2016, we have updated the fossil fuel emissions over the total period 2000–2015 to match the total global emissions used in GCP (Sect. ). These higher emissions lead to larger net carbon sinks, especially in the Northern Hemisphere. Following from the uncertainty estimate taken as the range of the different versions, we can state that the change between CTE2008–CTE2013 to CTE2014–CTE2016-FT has a significant effect on the resulting carbon flux estimates, which is a result of the used convective fluxes. The distribution of the sinks over the hemispheres shifted from the north to tropics and from the land to the oceans. With the updated convection, the land sink is especially decreased in the Northern Hemisphere, and the ocean sink is slightly increased in both the Northern and Southern hemispheres.