Conventional integration of Earth system and ocean models can accrue considerable computational expenses, particularly for marine biogeochemical applications. “Offline” numerical schemes in which only the biogeochemical tracers are time stepped and transported using a pre-computed circulation field can substantially reduce the burden and are thus an attractive alternative. One such scheme is the “transport matrix method” (TMM), which represents tracer transport as a sequence of sparse matrix–vector products that can be performed efficiently on distributed-memory computers. While the TMM has been used for a variety of geochemical and biogeochemical studies, to date the resulting solutions have not been comprehensively assessed against their “online” counterparts. Here, we present a detailed comparison of the two. It is based on simulations of the state-of-the-art biogeochemical sub-model embedded within the widely used coarse-resolution University of Victoria Earth System Climate Model (UVic ESCM). The default, non-linear advection scheme was first replaced with a linear, third-order upwind-biased advection scheme to satisfy the linearity requirement of the TMM. Transport matrices were extracted from an equilibrium run of the physical model and subsequently used to integrate the biogeochemical model offline to equilibrium. The identical biogeochemical model was also run online. Our simulations show that offline integration introduces some bias to biogeochemical quantities through the omission of the polar filtering used in UVic ESCM and in the offline application of time-dependent forcing fields, with high latitudes showing the largest differences with respect to the online model. Differences in other regions and in the seasonality of nutrients and phytoplankton distributions are found to be relatively minor, giving confidence that the TMM is a reliable tool for offline integration of complex biogeochemical models. Moreover, while UVic ESCM is a serial code, the TMM can be run on a parallel machine with no change to the underlying biogeochemical code, thus providing orders of magnitude speed-up over the online model.
The transport matrix method (TMM)
The TMM has been applied to a wide range of
problems, including simulating anthropogenic carbon uptake and radiocarbon by
the ocean
Despite these varied applications, a comprehensive evaluation of the TMM viz
a vis results produced by the corresponding online model has not yet appeared
in the published literature.
The TMM relies on the fact that the underlying partial differential equation for
transport of a passive tracer is linear with respect to
advection and diffusion. If the discrete (numerical) implementation is also linear,
the tracer time-stepping equation can be generally
written in matrix form as
The procedure for extracting the TMs is described in detail in
It is quite straightforward to implement the above procedure in an ocean
model with only minor modifications to the code, and we have done so with
UVic ESCM. However, in practice a number of complications can arise. For
example, ocean models sometimes use a non-linear advection scheme so as to
avoid over- and undershoots arising from sharp gradients in the tracer field.
In fact, UVic ESCM uses such a scheme, “flux-corrected transport” (FCT;
A second complication is from the time-stepping scheme, which in UVic ESCM
is leapfrog. The explicit horizontal advection and diffusion terms are also
sometimes staggered with respect to each other for stability. Both require
storing the tracer field at odd and even time steps. While this can be
replicated offline, in order to use a common scheme for all ocean models from
which TMs have been extracted (e.g. MITgcm variously uses Adams–Bashforth,
direct space–time discretization and other schemes), we combine horizontal
advection and diffusion into a single explicit transport matrix,
Lastly, UVic ESCM applies Fourier filtering in the zonal direction at high
latitudes to remove grid-scale noise. The efficiency of the TMM arises from
the fact that the discretized advection–diffusion operator has a limited
stencil, i.e. only couples nearby points, giving rise to a sparse matrix.
Fourier filtering on the other hand couples all points in the zonal
direction, greatly reducing the sparsity of the transport matrix and hence
the computational efficiency of the sparse matrix–vector products at the
heart of the TMM. While the cost of a sparse matrix–vector product is
implementation- and hardware-dependent and non-trivial to analyze
The neglect of polar filtering and staggering of advection and diffusion terms necessitates using, for stability, a slightly smaller time step in offline simulations with the TMM compared with the online model. In the latter, the default time step for ocean dynamics and tracer transport is 1.25 days, a choice dictated by the need to synchronize the ocean model with the atmospheric one (which takes two time steps per ocean time step). (The biogeochemical terms in UVic ESCM are time stepped internally within the biogeochemical module using a much smaller time step such that there are three biogeochemical steps per ocean step.) No change was made to this during extraction of the TMs; i.e. the model physics and active tracers were integrated using the default time step. Since the explicit TM is extracted as a tendency, the time step for offline explicit advection–diffusion can be subsequently set to any desired value. However, the implicit TM has a time step embedded within it. By default it would also be 1.25 days, but embedding a different time step is quite straightforward: during extraction of the implicit TM, we simply pass the desired time step as an argument to the subroutine that solves for implicit diffusion. We have found an offline time step of 8 h (28 800 s) to be a good compromise between stability and accuracy. It is also very similar to the biogeochemical time step of the online model (27 000 s).
Using the linear UW3 advection scheme, the coupled physical–biogeochemical
model was spun-up to equilibrium for 13 000 years with a fixed,
pre-industrial atmospheric CO
Dissolved inorganic carbon in the offline simulation
Annually averaged surface phytoplankton (top) and diazotroph
(bottom) biomass in mmol N m
Same as Fig.
Same as Fig.
Once the TMs have been extracted, biogeochemical tracers can be simulated
offline via Eq. (
To apply the TMM code to a particular biogeochemical problem essentially
requires providing a routine that takes as input vertical “profiles” of
tracer concentrations at a horizontal location at the current time step
(along with corresponding variables such as layer thickness, temperature,
wind speed, etc., at that location), and returns profiles of the biogeochemical
tendency term,
While the specific implementation of the wrapper will depend on the details
of the biogeochemical model, in general it performs three main tasks. First,
it copies required data from TMM arrays (that are passed as input arguments
to the wrapper routine) to those of the biogeochemical model. Second, it
calls the actual routine that computes biogeochemical source/sink terms
(
Same as Fig.
The offline biogeochemical model is forced with the relevant physical and biogeochemical fields taken from the equilibrated online model. In the present case, these are monthly mean wind speed, insolation, sea ice concentration, temperature, salinity, freshwater flux (evaporation, precipitation and runoff), and iron concentration. All fields, including the previously extracted transport matrices (also at monthly mean resolution), are linearly interpolated to the current time step before being applied. The offline model was integrated with a time step of 8 h for 5000 years to equilibrium, with monthly averages of various fields from the final year of this run used for comparison with the equilibrated online simulation.
Same as Fig.
Taylor diagrams for various simulated biogeochemical quantities in
the offline model compared with the online one. Different symbols and colors
correspond to different ocean basins. For each symbol plotted, the azimuthal
angle is the correlation coefficient between the offline and online field,
with a correlation coefficient of “1” lying on the
Zonally averaged surface biomass for phytoplankton (top),
diazotrophs (middle), and zooplankton (bottom) as a function of month and
latitude in the offline simulation (left column), online simulation (center),
and relative difference between the two (right). Relative error is calculated
as (UVIC_TM - UVIC_UW3)/UVIC_UW3. Units are mmol N m
Monthly mean surface phosphate, nitrate, and phytoplankton biomass as a function of calendar month for selected points. The map view indicates the location of the points as diamonds. Models are shown with colors: red is UVIC_UW3, green is UVIC_TM.
We first compare the annual-mean state of the online (UVIC_UW3) and offline
(UVIC_TM) simulations, taken from the final year of the corresponding
spin-up. A fully stable UVic ESCM simulation with annually repeating
seasonal forcing has no inter-annual variability. The surface DIC distribution (Fig.
Similar to DIC, differences in surface alkalinity values rarely exceed
5
Phosphate and nitrate distributions likewise show negligible differences
between models with the exception of polar regions and the water masses
closely associated with those regions (Figs.
Oxygen distributions in UVIC_TM are up to 50 mmol m
The above results are summarized in a spatially integrated manner via the
Taylor plots and associated tables shown in Fig.
In addition to the time–mean state it is important for any offline approach
to capture the seasonal cycle. Figure
The timing of seasonal maxima also appear well aligned between offline and
online runs when examined at several locations (Fig.
UVic ESCM is a serial code and thus unable to exploit more than one
computational core. With the biogeochemical component switched on the model
throughput on a typical Linux machine (ours has a 2.6 GHz Sandy Bridge-EP
processor) is about
This study investigates the extent to which the transport matrix method, a scheme for offline simulation of ocean biogeochemical tracers, can reproduce the corresponding online model, specifically an NPZD-type biogeochemical model embedded into UVic ESCM. While the focus is on a detailed comparison of the offline run with the online one, we also describe the mechanics of extracting transport matrices from UVic ESCM and coupling its biogeochemical model to the TMM framework. As the steps required for both aspects are quite general, this may be useful for researchers interested in applying the TMM to other ocean and biogeochemical models.
We show that the TMM version captures reasonably well both the time–mean
state and seasonal cycle of biogeochemical tracers of the online model. Small
discrepancies arise at high latitudes due to the absence of polar filtering
in the offline model. Differences also arise from the time averaging (monthly
in these experiments) of forcing fields and the circulation embedded in the
transport matrices, although the degree of averaging can be varied to suit
the situation. Phytoplankton are especially sensitive to this time averaging,
with the impact on most biogeochemical tracers being limited to the surface
ocean. These differences are generally much smaller than biases in the models
with respect to observations. While UVic ESCM is a serial
model, the TMM version can be run in parallel without any modifications to
the underlying code. Simulations performed with the TMM are thus orders of
magnitude faster making it possible to routinely perform long spin-ups of
UVic ESCM biogeochemistry in a few hours rather than weeks. A recent study
The specific version of the offline
biogeochemical model code, transport matrices and other data used for the
simulations described in this paper are available from
Here we discuss the changes due to switching from UVic ESCM's default,
non-linear flux-corrected transport scheme (UVIC_FCT) to the third-order linear
advection scheme (UVIC_UW3). As noted in Sect.
Meridional overturning in flux-corrected transport (UVIC_FCT) and third-order upwind advection (UVIC_UW3) online models. Positive (negative) values indicate clockwise (counterclockwise) circulation.
Background radiocarbon in online models UVIC_FCT and UVIC_UW3
compared with gridded observations from the GLODAP dataset
Potential temperature in online models UVIC_FCT and UVIC_UW3
compared with the World Ocean Atlas climatology
Salinity in online models UVIC_FCT and UVIC_UW3 compared with the
World Ocean Atlas climatology
Taylor diagrams for various simulated biogeochemical quantities in
comparison to observations (World Ocean Atlas climatology
Figure
The Taylor diagrams shown in Fig.
SK and KK wrote the paper (with comments by HD, IK, and AO), implemented the matrix interface and extraction, and performed the simulations. HD re-tuned the online model to use a linear advection scheme and assisted with interpretation. AO, IK and SK conceived the project.
The authors declare that they have no conflict of interest.
This work is a contribution to DFG-supported project SFB754 and to the research platforms of DFG cluster of excellence The Future Ocean. Khatiwala was supported in part by US NSF grant OCE 12-34971 and UK NERC grant NE/K015613/1. Computing resources (ark:/85065/d7wd3xhc) were provided by the Climate Simulation Laboratory at NCAR's Computational and Information Systems Laboratory, sponsored by the National Science Foundation and other agencies. We also gratefully acknowledge computer resources made available at Kiel University, technical consultation with Michael Eby and figure scripts by Andreas Schmittner.Edited by: Paul Halloran Reviewed by: three anonymous referees