Modeling atmospheric chemistry at fine resolution globally is computationally expensive; the capability to focus on specific
geographic regions using a multiscale grid is desirable.
Here, we develop, validate, and demonstrate stretched grids in the GEOS-Chem atmospheric chemistry model in its
high-performance implementation (GCHP).
These multiscale grids are specified at runtime by four parameters that offer users nimble control of the region that is refined
and the resolution of the refinement.
We validate the stretched-grid simulation versus global cubed-sphere simulations.
We demonstrate the operation and flexibility of stretched-grid simulations with two case studies that compare simulated tropospheric
NO2 column densities from stretched-grid and cubed-sphere simulations to retrieved column densities from the TROPOspheric
Monitoring Instrument (TROPOMI).
The first case study uses a stretched grid with a broad refinement covering the contiguous US to produce simulated columns that
perform similarly to a C180 (∼ 50 km) cubed-sphere simulation at less than one-ninth the computational expense.
The second case study experiments with a large stretch factor for a global stretched-grid simulation with a highly localized
refinement with ∼10 km resolution for California.
We find that the refinement improves spatial agreement with TROPOMI columns compared to a C90 cubed-sphere simulation of comparable
computational demands.
Overall, we find that stretched grids in GEOS-Chem are a practical tool for fine-resolution regional- or continental-scale
simulations of atmospheric chemistry.
Stretched grids are available in GEOS-Chem version 13.0.0.
Introduction
Global simulations of atmospheric chemistry are computationally demanding.
Chemical mechanisms in the troposphere typically involve more than 100 chemical species, emitted by anthropogenic and natural sources,
with production and loss by chemical reactions, and mixing through 3-D transport on all scales.
Typical global model resolutions are on the order of hundreds of kilometers and generally limited by the degree of model parallelism.
Massively parallel models such as GEOS-Chem in its high-performance implementation (GCHP; ) can
run on more than 1000 cores with a demonstrated capability of 50 km resolution.
The coarse resolution of global models can lead to systematic errors in applications when scales of variability finer than the model
resolution are relevant, such as vertical transport and scavenging by convective updrafts
, nonlinear chemistry such as NOx titration
, localized emission sources , a priori profiles for
satellite retrievals , and simulated concentrations for
population exposure estimates .
Grid refinement is commonly used for simulations that need to capture fine-scale modes of variability.
Here we implement grid refinement in GCHP using a technique that stretches the model grid to enhance its resolution in a user-defined
region, enabling massively parallel global multiscale simulations of atmospheric chemistry.
We validate the implementation, discuss key considerations for stretched-grid simulations, and demonstrate the capability.
The general approaches to grid refinement are nesting, adaptive grids, and grid-stretching.
Nesting broadly describes the use of a coarse global grid in combination with finer regional grids.
In one-way nesting, a coarse global simulation generates boundary conditions for one or more regional simulations
.
One-way nesting is simple yet effective when regional feedbacks on the global simulation are not required.
In two-way nesting, the coarse global simulation is dynamically coupled to the regional simulations to allow for feedbacks
.
Two-way nesting is more complex technically than one-way nesting, but the regional feedbacks from finely resolved chemical and
physical processes can improve global simulations of trace gases such as CO and O3.
One-way and two-way nesting are supported by the single-node version of GEOS-Chem “Classic”
, and by GEOS-Chem using Weather Research and Forecasting (WRF) meteorology
(WRF-GC; ).
An adaptive grid is a grid that supports dynamic refinement.
Adaptive grids allow the refinement to continuously target regions where simulation accuracy is the most sensitive to model resolution
.
Compared to static refinements, adaptive grids can better capture strong chemical gradients, which can improve the representation of
nonlinear chemistry and reduce numerical diffusion .
Adaptive grids are promising, but they are inherently complex and have not yet been used for global atmospheric chemistry simulations.
Stretched grids are global grids that are “stretched” by an analytic transform to enhance the grid resolution in a region
.
This stretching creates a global grid with a single refinement and smooth gradual changes in resolution.
A key advantage of grid-stretching is simplicity.
Stretching does not change the logical structure (topology) of the grid, so fundamental changes to the model structure are not required
and two-way coupling is inherent.
An additional benefit is that lateral boundary conditions are not required.
A drawback of grid-stretching, however, is that a stretched-grid simulation has a single refinement.
The first use of stretched grids for atmospheric chemistry simulations was in and ; more
recently, used a stretched grid for a fine-resolution simulation of aerosols in Japan, and
used a stretched grid for a fine-resolution simulation of surface-level O3 in Japan.
Several recent works set the stage for the development of grid-stretching in GEOS-Chem.
developed the grid-independent capability of GEOS-Chem.
developed the stretched grid capability for the GFDL Finite-Volume Cubed-Sphere Dynamical Core (FV3),
which is used to calculate advection in GCHP.
developed the capability for GEOS-Chem to operate on cubed-sphere grids in a distributed memory
framework for massive parallelization and to use the Model Analysis and Prediction Layer (MAPL; ) of the NASA
Global Modeling and Assimilation Office (GMAO) together with the Earth System Modeling Framework (ESMF; )
to couple model components.
GEOS-Chem version 12.5.0 added grid-independent emissions that produce consistent emissions regardless of the model grid
.
Most recently, MAPL version 2 of the NASA GMAO added stretched grid support.
The remainder of the paper is organized as follows.
Sect. describes GCHP and the gnomonic cubed-sphere model grid.
Sect. describes the stretching transform, which is based on the transform
and follows the methodology of .
Sect. discusses considerations for stretched-grid simulations and a simple procedure for choosing
an appropriate stretch factor.
Section describes the shared model configuration that is used for all of the simulations in this paper.
Section validates the stretched-grid implementation by a comparison of simulated oxidants and PM2.5
concentrations from stretched-grid and cubed-sphere simulations.
Finally, case studies in Sect. and demonstrate and explore grid-stretching in GCHP by considering
the application of comparing simulated NO2 columns with regional observations from TROPOspheric Monitoring Instrument (TROPOMI).
Development of stretched grids in GEOS-ChemGEOS-Chem in its high-performance implementation (GCHP)
We use GEOS-Chem version 13.0.0 in its high-performance implementation (GCHP; ).
GEOS-Chem, originally described in , simulates tropospheric–stratospheric chemistry by solving 3-D
chemical continuity equations.
GCHP uses MAPL and ESMF to facilitate the coupling of model components and use
of the high-performance computing (HPC) infrastructure.
The 3-D advection component is the GFDL Finite-Volume Cubed-Sphere Dynamical Core .
Columnar operators are used for columnar or local calculations such as convection and chemical kinetics.
Emissions are aggregated, parameterized, and computed with the Harmonized Emissions Component (HEMCO) described in
.
Stratospheric chemistry is simulated using the Unified Chemistry Extension (UCX) described in .
Offline meteorological data are from the Goddard Earth Observing System (GEOS) data assimilation system
.
Emissions and meteorological input data are regridded online by ESMF using the first-order conservative scheme originally described in
.
GCHP discretizes the atmosphere with a gnomonic cubed-sphere grid with levels extending from the surface to 1 Pa.
The cubed-sphere grid has several advantages over a regular latitude–longitude grid, stemming from its more uniform grid boxes that
benefit the parallelization and numerical stability of transport .
The horizontal resolution of a GCHP simulation is a key determinant of its computational demands.
An example of the horizontal grid is illustrated in Fig. .
A gnomonic cubed-sphere grid is a mosaic of six grids hereafter referred to as “faces”.
Each face is a logically square grid that is regularly spaced in a gnomonic projection centered on the face.
One of the six faces is highlighted in Fig. .
The position of the faces are fixed, and the center of the first face is 0∘ N, 10∘ W.
Hereafter, we refer to a gnomonic cubed-sphere grid as simply a “cubed sphere” or a “cubed-sphere grid”.
The horizontal resolution of a cubed sphere is dictated by its size, which is an even integer denoted with the notation CN
(e.g., C180); each face in the six-face mosaic is an N×N grid.
The computational demand of a GCHP simulation is proportional to the total number of grid boxes.
Table provides a comparison of cubed-sphere grids and conventional latitude–longitude grids.
We note for context that GEOS-Chem Classic can use a 2∘×2.5∘ or 4∘×5∘ global grid.
Offline meteorological data for GCHP are provided by the GMAO.
GCHP uses a local archive of the GEOS-FP or MERRA-2 data product.
GEOS-FP is a near real-time analysis product with a 0.25∘×0.3125∘ grid.
MERRA-2 is a reanalysis product with a 0.5∘×0.625∘ grid.
The effect of topography and surface type on transport is implicit in the meteorological data that drives transport.
Both data products have a 72-level terrain-following hybrid-sigma pressure grid that extends from the surface to 1 Pa.
GCHP uses a vertical grid that is identical to the meteorological data, and thus vertical regridding is not required.
Illustration of a C16 grid. A cubed-sphere grid always has six faces. One of the six faces is
highlighted for illustrative purposes. The highlighted face is a 16 × 16 grid
that is regularly spaced in a gnomonic projection centered on the face.
Characteristics of various cubed-sphere grids and global latitude–longitude grids.
a Number (no.) of grid boxes is listed for one vertical level. The GEOS-FP and MERRA-2 meteorological fields currently have 72 vertical levels.
b Here we define the resolution of a grid box as the square root of its area.
Grid-stretching procedure
The grid-stretching procedure in GCHP uses a simplified form of the transform for gnomonic cubed-sphere
grids, following the methodology of .
The Schmidt transform can be applied to any grid and effectively stretches the grid to increase its density in a region.
The procedure has two steps, starting with a standard cubed-sphere grid.
First, the grid is refined at the South Pole by remapping the grid coordinate latitudes with a modified Schmidt transform
ϕ′(ϕ)=arcsinD+sinϕ1+DsinϕwhereD=1-S21+S2,
where ϕ is an input latitude, ϕ′ is the output latitude, and S is a parameter called the “stretch factor”.
The stretch factor controls the strength of the operation, which effectively attracts the grid coordinates towards the South Pole
along meridians.
The second step is rotating the entire grid so that the refinement is repositioned to the desired region.
The user specifies a “target latitude”, Tϕ, and “target longitude”, Tθ.
The refinement is re-centered to these coordinates by rotating the grid.
Three stretched grids that illustrate the effect of the stretch factor (S) on stretching a C16 cubed sphere.
Local scaling is the relative change to a grid box's edge length induced by stretching.
Figure illustrates the effect of S on stretching a cubed-sphere grid.
A stretch factor greater than 1 causes stretching. Larger stretch factors cause more stretching and result in a finer and more
localized refinement.
The resolution at the center of the refinement is approximately S times finer than it was before stretching, and the
antipode resolution similarly is approximately S times coarser.
These relative changes are approximate since the Schmidt transform is continuous and the grid boxes have nonzero length edges.
The grids in Fig. illustrate three noteworthy features of stretched grids: (1) the changes in
resolution are smooth, (2) the refined domain diminishes as S increases, and (3) grid boxes outside the refined domain expand.
The cubed-sphere face at the center of the refined domain is called the “target face”.
The relative change to a grid box size from stretching can be quantified by “local scaling”.
This quantity represents the effect of grid-stretching at a given point.
For a stretch factor of S, the local scaling at a given point depends exclusively on how far that point is from the target coordinate.
Local scaling, L, can be derived from Eq. (), and expressed as
L(Θ;S)=1+cosΘ+S2(1-cosΘ)2S,
where Θ is the angular distance to the target point.
Appendix contains the derivation of Eq. ().
Figure shows local scaling as a function of distance, for stretch factors between 1 and 10.
Overlaid are dashed lines that show the lateral position of the face edges after stretching.
In the target face, local scaling is approximately 1/S and nearly constant.
The grid is refined up to the distance where L=1 and coarsened at greater distances.
Local scaling as a function of distance from the target point, for stretch factors in the range 1–10.
The dashed lines show the distance from the target point to the cubed-sphere face edges after stretching.
The lower dashed line is the distance to the center of the target face's edges, and the upper dashed line is the distance to
the center of the opposite face edges.
Four scalar parameters fully describe a stretched grid: the size of the cubed sphere, the stretch factor (S), the target latitude
(Tϕ), and the target longitude (Tθ).
These concise parameters are conceptually simple, precise, and give the user nimble control of the grid.
The combination of cubed-sphere size and stretch factor controls the grid resolution.
The target latitude and longitude specify the center of the refined domain.
Moderate stretch factors (e.g., 1.4–3.0) are suitable for broad refinements for continental-scale studies.
Large stretch factors (e.g., > 5.0) are suitable for localized refinements for regional-scale studies.
Choosing an appropriate stretch factor
Although the stretch factor is a well-defined parameter, appropriate values for atmospheric chemistry simulations will be
application-specific and moderately variable.
For computational efficiency, it is desirable to use the largest viable stretch factor to achieve the finest refinement for a given
cubed-sphere size.
However, larger stretch factors also result in a smaller refinement and in coarser resolutions outside the refined domain.
To determine the maximum suitable stretch factor for a given application, one should consider the size of the domain that should be
refined and the sensitivity of the study species to coarse resolutions outside the refinement.
A simple procedure for choosing a stretch factor, S, is choosing the maximum stretch factor subject to two constraints:
Constraining S by the size of the refined domain.
Local scaling is approximately constant and equal to 1/S throughout the target face.
Therefore, defining the refined domain as the region where resolution is enhanced by a factor of
∼S, the target face is a reasonable approximation of the refined domain.
For a target face with width greater than wtf, S must satisfyS≤0.414cotwtf/4rE,where rE is Earth's radius.
Here, we define target-face width, wtf, as the edge-to-edge distance across the center of the face.
For example, for a refined domain with a diameter of at least 3000 km, the target face should be at least 3000 km wide, and thus the
stretch factor should be less than or equal to 3.5.
Alternatively, one could inspect Fig. to find the distance from the target point to the edge
of the target face for a given value of S.
Constraining S by a maximum and minimum resolution.
The target resolution is S2 times finer than its antipode's resolution; therefore, a constraint for S based on the desired
maximum resolution, Rmax, and minimum resolution, Rmin, isS≤Rmax/Rmin.The minimum resolution, Rmin, is an important consideration.
Rmin imposes a limit on the coarsest resolution outside of the refined domain.
Therefore, the choice of an appropriately fine Rmin can reduce potential bias in species in the coarse grid
outside of the refined domain from affecting the study species in the refined domain.
For example, for a maximum and minimum resolution comparable to C360 and C24 cubed spheres, S≤3.9, or for a maximum and
minimum resolution comparable to C360 and C48 cubed spheres, S≤2.7.
Once the constraints for S are determined, one can choose S and the grid size for their simulation.
For example, for a stretched grid with a refined domain with a diameter greater than 3000 km, a maximum resolution of C360, and a
minimum resolution of C24, the constraints would be S≤3.5 and S≤3.9.
To determine the grid size and stretch factor for a simulation from these constraints, first, assume S=3.5 as an initial value.
For a maximum resolution comparable to a C360 grid with S=3.5, the grid size would be C102.9 (360/3.5=102.9).
Since the grid size must be an even integer, one should round up to C104 and choose S=3.46 (360/104=3.46).
The single refinement and the expansion of grid boxes outside the refined domain are important limitations of grid-stretching.
The coarse grid outside the refined domain is susceptible to resolution-dependent biases in O3 and CO
, which could influence the representation of chemical processes in the
refined domain.
Therefore, the choice of an appropriately fine minimum resolution is important (Rmin in constraint 2).
Generally, stretched-grid simulations are well suited for applications that are principally sensitive to emissions and physical
processes in the refined domain.
For example, stretched-grid simulations are well suited for regional studies of boundary layer concentrations of short-lived species.
Applications such as evaluations of the global tropospheric O3 budget are better suited for standard cubed-sphere simulations.
Model configuration
The simulations in this paper use a shared model configuration of GCHP version 13.0.0-alpha.3.
We use default emissions for all species.
Table summarizes the emission inventories that represent sources of NOx in the simulations.
Briefly, we use monthly anthropogenic emissions based on the National Emission Inventory (NEI) for 2011 with updated annual scaling
factors to account for changes in annual totals since 2011.
In 2019, the scaling factor for NO was 0.61.
Biomass burning emissions are from the Global Fire Emissions Database version 4 (GFED4; ).
Aircraft emissions are from the Aviation Emissions Inventory Code (AEIC; ).
We use inventories for NOx emissions from soil microbial activity and lightning calculated offline at the native resolution
of the meteorology .
Offline soil NOx emissions are based on the scheme described in , and offline lightning
NOx emissions are based on the scheme described in .
For meteorological data, we use GEOS-FP data because it offers superior horizontal resolution data for advection input variables
(0.25∘× 0.3125∘).
All simulations use a 10 min time step for chemistry and a 5 min time step for transport.
Simulations are named according to their grid. Standard cubed-sphere simulations are named by their resolution with the suffix
“-global”, referring to their resolution being quasi-uniform globally.
For example, a standard C180 cubed-sphere simulation is named “C180-global”. Stretched-grid simulations are named according to
their refinement effective resolution, with a suffix denoting the region that is refined.
For example, a stretched-grid simulation with an effective resolution of C180 in the contiguous US is named “C180e-US”.
“C180e” refers to the stretched-grid refinement being comparable to the resolution of a C180 cubed-sphere grid.
Emissions inventories in GEOS-Chem 13.0.0 that represent NOx sources in the simulations conducted in this paper.
Source typeInventoryResolutionReference/notesAnthropogenic USNEI-20110.1∘× 0.1∘Annual totals are updated for 2018/19a AsiaMIX0.25∘× 0.25∘ GlobalCEDSb0.5∘× 0.5∘Lightning–0.25∘× 0.3125∘Soil–0.25∘× 0.3125∘Biomass burningGFED40.25∘× 0.25∘AircraftAEIC1.0∘× 1.0∘ShippingCEDS0.5∘× 0.5∘
a Monthly fluxes, including diurnal and weekday–weekend variations and vertical allocations,
based on criteria pollutants National Tier 1 from https://www.epa.gov/air-emissions-inventories/air-pollutant-emissions-trends-data (last access: 8 May 2020).
b Community Emissions Data System (CEDS)
Validating the stretched-grid capability
Next, we test the implementation of stretched grid by comparing the concentrations of oxidants and PM2.5 from stretched-grid and cubed-sphere simulations.
Prior tests were also conducted to identify and fix several technical errors in some component capabilities.
We choose the contiguous US as the domain for this comparison and a standard C96 cubed-sphere simulation to serve as the control simulation (C96-global).
The stretched-grid simulation, C96e-NA, has a grid size of C48 and stretching parameters
S=2.4, Tϕ=35∘ N, and Tθ=96∘ W. The target point
was chosen so that the target face approximately encompassed the populous regions of North America.
The stretch factor was chosen so that the average resolution of C96e-NA was equal to the
average resolution of C96-global in the contiguous US;
we note that the stretch factor is 2.4, rather than 2.0, because the US is a region where
the standard cubed-sphere grid has a finer resolution than its global average, as shown in Appendix .
Figure compares the resolution of C96e-NA and
C96-global grids.
A consequence of the similar resolution of the C96-global and C94-global grids is that their grid boxes
have little overlap; this makes their comparison sensitive to the precision of upscaling emissions
(aliasing effects, i.e., differences in upscaled emissions like NO point sources from differences in
how the grids cover a region).
To calibrate an expectation for these differences, we compare C96-global to a second standard cubed-sphere
simulation, C94-global.
We choose C94-global because the similar resolution minimizes the frequency that its grid boxes overlap with the C96-global grid boxes.
Therefore, the comparison of C94-global and C96-global isolates differences from aliasing effects.
The frequency of grid box overlap between C96-global and C94-global is analogous to a beat frequency of 2; the regions where
overlap occurs is near the edges and across the center lines of the faces.
Comparison of C96-global and C96e-NA grids. The top panels show the variability of resolution for both grids.
The bottom panel shows the grid face edges for C96-global and C96e-NA, with the 125, 250, and 500 km
resolution contours for C96e-NA.
Figure compares C96e-NA with C96-global and C94-global with C96-global.
These comparisons are for the fourth simulation month, to accommodate relaxation
time for CO and O3 (the simulations started 1 June 2018 and ran through 1 October 2018).
All three simulations used an identical model configuration, apart from the grid, which is described in Sect. .
The scatter near the surface is caused by differences in the spatial allocation of emissions on different grids (the precision of
upscaling).
Figure shows that simulated averages from C96e-NA are consistent with C96-global and that
their differences are comparable to the differences between two similar cubed-sphere simulations.
We note a small low bias in ozone above 400 hPa associated with nonlinear chemical sensitivity to coarse grid boxes outside of the
refined region.
The C96e-NA grid resolution is coarser than 250 km in most of Asia, while the C96-global grid is finer than 125 km
(Fig. ).
A finer grid size or a smaller stretch factor could be used to increase resolution outside the refinement if this type of bias is of
concern for a stretched-grid application.
Comparison of simulated concentrations from C96e-NA with C96-global, and C94-global with C96-global.
Points are 1-month time averages for each grid box in the contiguous US for the fourth simulation month (September 2018).
The right column gauges expected differences due to the precision of upscaling emissions to different grids.
Concentrations in the lowermost model level are shown in purple.
Stretched-grid case studies
Next, we demonstrate stretched-grid simulations with GCHP by conducting case studies in Sect. and .
The applications we consider are regional comparisons of simulated tropospheric NO2 columns with observations.
NO2 is chosen because it is a well-measured species, and the sensitivity of its simulated concentrations to model resolution and
local chemical and physical processes make it a prime example of an application that is well suited for a stretched-grid simulation.
Here we explore two of the primary reasons one might use the stretched-grid capability: (1) for a computationally efficient regional
simulation and (2) to realize a finer resolution than is otherwise possible.
Section considers a comparison of columns in the contiguous US; columns are simulated at ∼ 50 km resolution, with
stretched-grid and standard cubed-sphere simulations, to examine the ability of a stretched-grid simulation to produce similar results
to a cubed-sphere simulation at a lesser computational expense.
Section experiments with the use of a very large stretch factor for a simulation targeting California with ∼ 10 km
resolution and modest computational requirements.
We compare simulated tropospheric NO2 column densities to retrieved column densities from TROPOMI.
TROPOMI is an instrument onboard the Sentinel-5P satellite launched in 2018 into a sun-synchronous orbit with a local overpass time
of 13:30 LST (local solar time).
TROPOMI includes ultraviolet and visible band spectrometers.
The retrieved NO2 column densities have 3.5 × 5.5 km2 resolution and are calculated using a modified version of the Dutch
OMI NO2 (DOMINO) retrieval algorithm .
We include observations with retrieved cloud fractions less than 10 %.
Retrieved NO2 column densities are sensitive to the a priori profiles used to calculate the air mass factors
.
To avoid spurious differences from the retrieval a priori profiles when comparing simulated and retrieved NO2 column densities, we
recalculate the air mass factors with the mean simulated relative vertical profiles (shape factors) from the stretched-grid simulations
following the approach described in and .
Evaluations of TROPOMI NO2 columns show good correlation with ground-based measurements with a small low bias
.
Comparison of C180-global and C180e-US grids. The top panels show the variability of resolution for both grids.
The bottom panel shows the grid face edges for C180-global and C180e-US, with the 100, 250, and 500 km
resolution contours for C180e-US.
A stretched-grid simulation with a moderate stretch factor
Here we consider a comparison of simulated and observed NO2 columns in the contiguous US.
We compare simulated columns with ∼ 50 km resolution from stretched-grid and standard cubed-sphere simulations with observations from TROPOMI.
Our focus is on the technical ability to use a stretched grid for an efficient fine-resolution simulation of the columns
compared to using a standard cubed sphere for this purpose.
The contiguous US is a large refinement domain, and thus a moderate stretch factor is appropriate.
This case is intended to be representative of a typical case where grid refinement is used in global models.
Mean tropospheric NO2 column densities from C180-global, C180e-US, C60-global, and TROPOMI observations
for July 2019. Simulated means only include points where TROPOMI observations were available.
TROPOMI columns shown here use shape factors from C180e-US. TROPOMI columns with shape factors from C180-global
were nearly identical.
An annotated map of the contiguous US is provided in Fig. .
For the cubed-sphere simulation, we use a C180 grid and refer to it as C180-global.
For the stretched-grid simulation, we use a stretch factor of S=3.0, a grid size of C60, and a target point at 36∘ N, 99∘ W.
We refer to the stretched-grid simulation as C180e-US.
The stretching parameters are chosen so that the target face approximately encompasses the contiguous US.
Figure shows the resolution of the two simulation grids.
In the contiguous US, the average resolution of C180-global is 47.9 km, and the average resolution of C180e-US is 56.6 km.
The total number of grid boxes in C180e-US is 9 times fewer than the number of grid boxes in C180-global
(cf. C60 and C180 grids in Table ).
Both simulations use an identical model configuration, apart from the grid.
The model configuration is described in Sect. .
To examine C180e-US without grid-stretching, we conduct a third simulation, C60-global.
The 2-week timing test results comparing the computational expense of C180-global, C180e-US, and C60-global.
C180-globalC180e-USC60-globalNumber of cores used (CPUs)3844848Wall time Chemistry(hours)21.87.07.2Dynamics(hours)3.11.71.7Data input(hours)0.51.80.3Other(hours)0.30.20.2Total wall time (hours)25.710.79.4Total CPU time (days)41121.318.7Model throughput (days/day)*13.131.435.7
* Simulation days per 24 wall hours.
Comparison of C90-global and C900e-CA grids. The top panels show the variability of resolution for both grids.
The bottom panel shows the grid face edges for C90-global and C900e-CA, with the 100, 500, and 1000 km
resolution contours for C900e-CA.
Mean tropospheric NO2 column densities from C90-global, C900e-CA, and TROPOMI observations for July 2019.
Simulated means only include points where TROPOMI observations were available.
TROPOMI columns shown here use shape factors from C900e-CA.
An annotated map of California is provided in Fig. .
Figure shows tropospheric NO2 column densities from TROPOMI, C180-global, C180e-US, and
C60-global for the US in July 2019.
All three simulations included a 1-month spinup.
An annotated map of the contiguous US is provided in Fig. .
The TROPOMI columns have high NO2 concentrations over major cities and low NO2 concentrations in rural and remote regions.
Simulated NO2 column densities from C180-global and C180e-US are consistent throughout the domain and generally capture
the plumes over cities and the low concentrations in rural and remote areas.
Small differences, like those seen near Four Corners and Denver, can be attributed to differences from upscaling the emissions to the
simulation grids (i.e., aliasing).
In the case of the differences near Four Corners, emissions from natural gas production and power plants in the region have a spatial
scale that is finer than the simulation grids.
C60-global failed to resolve the local enhancements over major cities; a comparison with C180e-US highlights the
effectiveness of grid-stretching.
The computational demands of C180e-US were significantly less than C180-global and comparable to C60-global.
Table gives timing test results for C180-global, C180e-US, and C60-global.
In terms of total computational workload, the total CPU time for C180e-US was ∼ 19 times less than C180-global.
The improvement is better than one would approximate by simply considering the reduction in the total number of grid boxes because
the parallelization overhead is reduced as well.
In terms of model throughput, C180e-US was 2.4 times faster than C180-global, despite using 8 times fewer cores.
The total CPU time for C180e-US was 15 % greater than C60-global.
The slight increase of data input cost in C180e-US is suspected to be caused by a load imbalance in online input regridding in
ESMF.
Work to mitigate this load imbalance is underway.
A stretched-grid simulation with a large stretch factor
A large stretch factor creates a grid with a strong but localized refinement.
Here, we experiment with using a large stretch factor to simulate NO2 columns in California at ∼ 10 km resolution.
Simulated NO2 columns in California are known to be sensitive to model resolution because of significant variability in emissions
and topography .
This case study is intended to probe the computational ability for very fine simulations and the feasibility of using a highly
stretched grid for an application that is well-suited for stretching.
For the stretched-grid simulation, we choose a grid size of C90 with stretching parameters S=10,
Tϕ=37.2∘ N, and Tθ=119.5∘ W.
We refer to this simulation as C900e-CA.
The stretching parameters were chosen so that the target face approximately encompassed California.
For comparison, we also conduct a cubed-sphere simulation with a C90 grid, which we refer to as C90-global.
C90-global is effectively C900e-CA without grid-stretching.
Both simulations use an identical model configuration, apart from the grid.
Figure compares the resolution of the two simulation grids.
The average resolution of C900e-CA in California is 11.2 km.
To understand the variability of the C900e-CA grid resolution it is useful to consider the local scaling at various locations.
For example, New York is approximately 4000 km from C900e-CA's target point.
Figure shows that for S=10, the local scaling at 4000 km is close to 1.
Equivalently, substituting S=10 and Θ=4000 km/rE in Eq. () gives L=1.04.
Therefore, the resolution of C900e-CA in New York is similar to a standard C90 cubed sphere.
This can be confirmed in Fig. .
The combination of a large stretch factor (S=10) and moderate grid size (C90) causes some grid boxes to become very coarse.
Grid boxes in the Indian Ocean and southeastern Africa are coarser than 1000 km.
These coarse grid boxes will result in an errors in O3 production and an overestimation of stratosphere–troposphere exchange
.
Therefore, the C900e-CA grid would not be suitable for applications sensitive to O3 or CO in the mid-troposphere and upper troposphere.
This underscores that selection of an appropriate stretched grid is application specific.
C900e-CA leverages the fine spatial resolution of anthropogenic NO emissions data available for the US; the NEI-2011
inventory has a 0.1∘× 0.1∘ grid (∼ 9 km).
C900e-CA uses meteorological data from the GEOS-FP data product with a spatial resolution of
0.25∘× 0.3125∘ (∼ 25 km).
We expect some of the detailed orographic and coastal effects in California to be missed.
C900e-CA identifies a need for even finer resolution of meteorological data, for which there is ongoing work in the
GCHP–GMAO community.
In the regions where C900e-CA's grid is finer than the GEOS-FP data, the data are conservatively downscaled (online
by ESMF).
We evaluate the effect of using downscaled meteorological data on the simulated NO2 columns in
Appendix and find no significant ill effects for our application.
Figure shows tropospheric NO2 column densities from TROPOMI, C900e-CA, and C90-global
for California in July 2019.
Both simulations used a 1-month spinup.
An annotated map of California is provided in Fig. .
The TROPOMI columns have significant fine-scale variability throughout California; high concentrations over Los Angeles and in the San
Francisco Bay Area; and smaller high-concentration features over Sacramento, Fresno, and Bakersfield.
C900e-CA resolves many of the fine-scale spatial features seen in the TROPOMI columns, including the small high-concentration
features over Sacramento, Fresno, and Bakersfield.
The coarse resolution of C90-global fails to resolve most spatial features seen in the TROPOMI and C900e-CA columns
and significantly underestimates high concentrations, except in Los Angeles (LA).
The underestimation of high concentrations in C90-global, such as in the San Francisco Bay Area, is associated with the averaging
of fine-scale urban emissions over the coarse grid boxes.
A subtle feature in the TROPOMI columns is the strong gradient along the LA Basin perimeter.
Neither simulation captures this gradient well, and LA's plume spuriously spreads into the Mojave Desert.
The shallow inversion that prevents the plume from rising over the mountains might be better represented in finer-resolution
meteorological inputs that could better resolve coastal dynamics.
Nonetheless, C900e-CA demonstrates a pronounced improvement over C90-global at resolving the challenging heterogeneity of
California with similar computational expense (within 17 %).
Conclusions
Fine-resolution simulations of atmospheric chemistry are necessary to capture fine-scale modes of variability such as localized sources,
nonlinear chemistry, and boundary layer processes, but fine-resolution simulations have been impeded by computational expense.
This work developed stretched grids for the high-performance implementation of GEOS-Chem (GCHP).
The capability was validated against global cubed-sphere simulations.
Applications were probed with case studies that compared simulated concentrations with observations from the TROPOMI satellite
instrument.
Stretched grids enable multiscale grids in GCHP while complementing the other multiscale grid methods that are available in
GEOS-Chem implementations.
A primary benefit of grid-stretching is ease of use.
The refinement is flexible and controlled by four simple runtime parameters.
Stretched grids operate naturally in GCHP, so switching between a stretched grid and cubed-sphere grid is seamless.
Stretched-grid simulations are stand-alone simulations that do not require any pregenerated or dynamically coupled boundary conditions.
Compared to nested grids, the main disadvantages of grid-stretching are that there is a single refinement and that one cannot control
the refined domain, refinement resolution, and global resolution independently.
Stretched-grid simulations can be used for regional- or continental-scale simulation purposes.
Generally, stretch factors in the range 1.4–4.0 are applicable for large refined domains.
Higher stretch factors can be used for simulations at very fine resolution for regional-scale applications.
To aid in choosing an appropriate stretch factor, we propose a simple procedure based on choosing the maximum stretch factor subject
to two constraints: the size of the refined domain and the maximum and minimum resolution.
Computationally, stretched-grid simulations are capable of unprecedented spatial resolutions for GEOS-Chem.
Fine-scale emissions, with accurate spatial and temporal variability, and meteorological inputs with fine horizontal, temporal, and
vertical resolution are needed to fully exploit these newly achievable resolutions.
Stretched grids are publicly available and ready for scientific application in GEOS-Chem version 13.0.0.
Variability of cubed-sphere grid resolution
Figure shows the variability of a C180 cubed-sphere grid's resolution. The coarsest resolutions are at the
center of faces, and the finest resolutions are at the corners of the faces.
The average resolution of a cubed-sphere grid can be approximated by AE/6×N×N, where AE
is the surface area of Earth (∼510×106 km2) and N is the size of the cubed sphere (e.g., N=180 for a C180 grid).
The average resolution of a C180 grid is 51 km.
The resolution at the center of the face is 62 km, and the resolution at the corner of the face is 41 km.
Variability of a C180 cubed sphere's resolution.
Derivation of local scaling
Local scaling, L, is the relative change to a grid box's length from grid stretching.
Consider a line segment that follows a meridian. The line segment starts at y and has length Δy.
After remapping the line segment with the Schmidt transform (Eq. ), the line segment starts at y′ and has
length Δy′.
The limit of the segment's local scaling as Δy→0 is equal to the derivative of the Schmidt transform:
L(y′;S)=limΔy→0Δy′Δy=limΔy→0ϕ′(y+Δy)-ϕ′(y)Δy=ddyϕ′(y).
The derivative of the Schmidt transform is
ddyϕ′(y)=2SS2(1-siny)+siny+1,
and thus the local scaling at y′ is
L(y′;S)=2SS2(1-siny)+siny+1.
We can obtain the local scaling at y, rather than y′, by substituting the inverse of the Schmidt transform into
Eq. ().
This gives
L(y;S)=S2(1+siny)-siny+12S.
Finally, we can generalize Eq. () so it is a function of arclength from the target.
The center of the refinement after applying Eq. () is at the South Pole.
The arclength from y to the South Pole is Θ=y-(-π/2). Substituting y=Θ-π/2 into Eq. () gives
L(Θ;S)=S2(1-cosΘ)+cosΘ+12S.
Maps of locations discussed in the text
Figure shows a map of the contiguous US.
Locations that are discussed in the text and locations with notable features in Fig. are marked.
Figure shows a map of California. Locations that are discussed in the text and locations with notable features in
Fig. are marked.
Map of the contiguous US. Locations discussed in the text and locations with notable features in Fig.
are marked.
Map of the California. Locations discussed in the text and locations with notable features in
Fig. are marked.
Evaluation of interpolating meteorological inputs in California NO2 simulation
Section demonstrates the use of a stretched-grid simulation for simulating tropospheric NO2 column densities
in California.
The simulation, C900e-CA, has an average resolution of 11.2 km in California, which is approximately twice as fine as the input
meteorological data (0.25∘× 0.3125∘ grid, or ∼ 25 km resolution).
To check that interpolating meteorological data in C900e-CA does not have significant unexpected consequences, we downscaled
the data to a 0.5∘× 0.625∘ grid (∼ 50 km) and conducted a pair of stretched-grid simulations with ∼ 25 km
resolution; one used the 0.25∘× 0.3125∘ data, and the other used the downscaled 0.5∘× 0.625∘ data.
The NO2 columns from these simulations are shown in Fig. D1 (second row), along with the columns from TROPOMI and C900e-CA
(first row).
The grid for these simulations was a C60 cubed sphere with a stretch factor of S=6 (∼ 25 km) and a target point of
37.2∘ N, 119.5∘ W (same as C900e-CA).
Both used a 1-month spinup.
Comparison of the lower panels in Fig. shows that interpolating the 0.5∘× 0.625∘
meteorological inputs to the ∼ 25 km grid has little effect on the simulated NO2 columns.
This suggests the consequences of interpolating meteorological data in C900e-CA are minor for our application and that the
variability is driven by the resolution of emissions data (∼ 9 km resolution).
Mean tropospheric NO2 column densities from TROPOMI, C900e-CA, C360e-CA with 0.25∘× 0.3125∘
meteorological data, and C360e-CA with meteorological data downscaled to 0.5∘× 0.625∘.
Code availability
GEOS-Chem is an open source project distributed under the MIT License at https://github.com/geoschem/GCHP (last access: 28 June 2021). The exact version of GCHP used in this paper was 13.0.0-alpha.3 and is archived on Zenodo (10.5281/zenodo.4317978, ).
Data availability
The simulations in this paper use default GEOS-Chem 13.0 emissions and meteorological data that can be obtained from the GEOS-Chem input data repository: http://geoschemdata.wustl.edu/ExtData/ (last access: 4 October 2021). The instructions for downloading the input data for GEOS-Chem 13.0 from the data repository are given in the GEOS-Chem 13.0 User Guide (https://gchp.readthedocs.io, last access: 8 October 2021).
Author contributions
LB implemented and validated the stretched-grid capability in GCHP.
RVM, DJJ, and SP provided project oversight and top-level design.
MJC reprocessed TROPOMI columns with shape factors from simulations.
EWL, LB, and SDE performed MAPL 2 upgrade in GCHP.
BMA and TLC developed stretched-grid capability in MAPL 2.
WMP developed FV3.
HW, JL, LTM, and JM developed grid-independent emissions for GEOS-Chem, and CAK developed HEMCO.
LB wrote the manuscript.
All authors contributed to manuscript editing and revisions.
Competing interests
The authors declare that they have no conflict of interest.
Disclaimer
Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Acknowledgements
We are grateful to the Research Infrastructure Service (RIS) at Washington University in St. Louis for providing computing resources. The meteorological data (GEOS-FP) used in this study have been provided by the Global Modeling and Assimilation Office (GMAO) at NASA Goddard Space Flight Center. We thank the two anonymous reviewers for their constructive comments and suggestions.
Financial support
This research has been supported by the Earth Sciences Division (grant no. AIST-18-0011) and the Natural Sciences and Engineering Research Council of Canada (grant no. RGPIN-2019-04670).
Review statement
This paper was edited by Fiona O'Connor and reviewed by two anonymous referees.
ReferencesAllen, D., Pickering, K., Stenchikov, G., Thompson, A., and Kondo, Y.: A
three-dimensional total odd nitrogen (NOy) simulation during SONEX using a
stretched-grid chemical transport model, J. Geophys. Res.-Atmos., 105, 3851–3876, 2000.Bey, I., Jacob, D. J., Yantosca, R. M., Logan, J. A., Field, B. D., Fiore,
A. M., Li, Q., Liu, H. Y., Mickley, L. J., and Schultz, M. G.: Global
modeling of tropospheric chemistry with assimilated meteorology: Model
description and evaluation, J. Geophys. Res.-Atmos.,
106, 23073–23095, 10.1029/2001JD000807, 2001.Boersma, K., Braak, R., and van der A, R. J.: Dutch OMI NO2 (DOMINO) data
product v2. 0, Tropospheric Emissions Monitoring Internet Service
on-line documentation,
available at: http://www.temis.nl/docs/OMI_NO2_HE5_2.0_2011.pdf (last access: 5 July 2020), 2011.Boersma, K. F., Eskes, H. J., Richter, A., De Smedt, I., Lorente, A., Beirle, S., van Geffen, J. H. G. M., Zara, M., Peters, E., Van Roozendael, M., Wagner, T., Maasakkers, J. D., van der A, R. J., Nightingale, J., De Rudder, A., Irie, H., Pinardi, G., Lambert, J.-C., and Compernolle, S. C.: Improving algorithms and uncertainty estimates for satellite NO2 retrievals: results from the quality assurance for the essential climate variables (QA4ECV) project, Atmos. Meas. Tech., 11, 6651–6678, 10.5194/amt-11-6651-2018, 2018.Cooper, M. J., Martin, R. V., McLinden, C. A., and Brook, J. R.: Inferring
ground-level nitrogen dioxide concentrations at fine spatial resolution
applied to the TROPOMI satellite instrument, Environ. Res.
Lett., 15, 104013, 10.1088/1748-9326/aba3a5, 2020.
Courtier, P. and Geleyn, J.-F.: A global numerical weather prediction model
with variable resolution: Application to the shallow-water equations,
Q. J. Roy. Meteor. Soc., 114, 1321–1346, 1988.Davis, D. D., Grodzinsky, G., Kasibhatla, P., Crawford, J., Chen, G., Liu, S.,
Bandy, A., Thornton, D., Guan, H., and Sandholm, S.: Impact of ship emissions
on marine boundary layer NO x and SO 2
Distributions over the Pacific Basin, Geophys. Res. Lett., 28,
235–238, 10.1029/2000GL012013, 2001.Eastham, S. D., Weisenstein, D. K., and Barrett, S. R.: Development and
evaluation of the unified tropospheric–stratospheric chemistry extension
(UCX) for the global chemistry-transport model GEOS-Chem, Atmos.
Environ., 89, 52–63, 10.1016/j.atmosenv.2014.02.001, 2014.Eastham, S. D., Long, M. S., Keller, C. A., Lundgren, E., Yantosca, R. M., Zhuang, J., Li, C., Lee, C. J., Yannetti, M., Auer, B. M., Clune, T. L., Kouatchou, J., Putman, W. M., Thompson, M. A., Trayanov, A. L., Molod, A. M., Martin, R. V., and Jacob, D. J.: GEOS-Chem High Performance (GCHP v11-02c): a next-generation implementation of the GEOS-Chem chemical transport model for massively parallel applications, Geosci. Model Dev., 11, 2941–2953, 10.5194/gmd-11-2941-2018, 2018.Feng, X., Lin, H., and Fu, T.-M.: WRF-GC: online two-way coupling of WRF and GEOS-Chem for regional atmospheric chemistry modeling, EGU General Assembly 2020, Online, 4–8 May 2020, EGU2020-5165, 10.5194/egusphere-egu2020-5165, 2020Fox-Rabinovitz, M., Côté, J., Dugas, B., Déqué, M., and McGregor, J. L.:
Variable resolution general circulation models: Stretched-grid model
intercomparison project (SGMIP), J. Geophys. Res., 111,
D16104, 10.1029/2005JD006520, 2006.Fox-Rabinovitz, M., Cote, J., Dugas, B., Deque, M., McGregor, J. L., and
Belochitski, A.: Stretched-grid Model Intercomparison Project: decadal
regional climate simulations with enhanced variable and uniform-resolution
GCMs, Meteorol. Atmos. Phys., 100, 159–178,
10.1007/s00703-008-0301-z, 2008.Freitas, S. R., Longo, K. M., Chatfield, R., Latham, D., Silva Dias, M. A. F., Andreae, M. O., Prins, E., Santos, J. C., Gielow, R., and Carvalho Jr., J. A.: Including the sub-grid scale plume rise of vegetation fires in low resolution atmospheric transport models, Atmos. Chem. Phys., 7, 3385–3398, 10.5194/acp-7-3385-2007, 2007.
Garcia-Menendez, F. and Odman, M. T.: Adaptive grid use in air quality
modeling, Atmosphere, 2, 484–509, 2011.
Garcia-Menendez, F., Yano, A., Hu, Y., and Odman, M. T.: An adaptive grid
version of CMAQ for improving the resolution of plumes, Atmos. Pollut.
Res., 1, 239–249, 2010.Goldberg, D. L., Lamsal, L. N., Loughner, C. P., Swartz, W. H., Lu, Z., and Streets, D. G.: A high-resolution and observationally constrained OMI NO2 satellite retrieval, Atmos. Chem. Phys., 17, 11403–11421, 10.5194/acp-17-11403-2017, 2017.Goto, D., Dai, T., Satoh, M., Tomita, H., Uchida, J., Misawa, S., Inoue, T., Tsuruta, H., Ueda, K., Ng, C. F. S., Takami, A., Sugimoto, N., Shimizu, A., Ohara, T., and Nakajima, T.: Application of a global nonhydrostatic model with a stretched-grid system to regional aerosol simulations around Japan, Geosci. Model Dev., 8, 235–259, 10.5194/gmd-8-235-2015, 2015.Griffin, D., Zhao, X., McLinden, C. A., Boersma, F., Bourassa, A., Dammers, E.,
Degenstein, D., Eskes, H., Fehr, L., Fioletov, V., Hayden, K., Kharol, S. K.,
Li, S., Makar, P., Martin, R. V., Mihele, C., Mittermeier, R. L., Krotkov,
N., Sneep, M., Lamsal, L. N., Linden, M. t., Geffen, J. v., Veefkind, P., and
Wolde, M.: High‐Resolution Mapping of Nitrogen Dioxide With
TROPOMI: First Results and Validation Over the Canadian Oil
Sands, Geophys. Res. Lett., 46, 1049–1060,
10.1029/2018GL081095, 2019.Harris, L. M., Lin, S.-J., and Tu, C.: High-Resolution Climate
Simulations Using GFDL HiRAM with a Stretched Global Grid,
J. Climate, 29, 4293–4314, 10.1175/JCLI-D-15-0389.1, 2016.Heckel, A., Kim, S.-W., Frost, G. J., Richter, A., Trainer, M., and Burrows, J. P.: Influence of low spatial resolution a priori data on tropospheric NO2 satellite retrievals, Atmos. Meas. Tech., 4, 1805–1820, 10.5194/amt-4-1805-2011, 2011.Hill, C., DeLuca, C., Balaji, Suarez, M., and Da Silva, A.: The
Architecture of the Earth System Modeling Framework, Comput.
Sci. Eng., 6, 18–28, 10.1109/MCISE.2004.1255817, 2004.Hudman, R. C., Moore, N. E., Mebust, A. K., Martin, R. V., Russell, A. R., Valin, L. C., and Cohen, R. C.: Steps towards a mechanistic model of global soil nitric oxide emissions: implementation and space based-constraints, Atmos. Chem. Phys., 12, 7779–7795, 10.5194/acp-12-7779-2012, 2012.Ialongo, I., Virta, H., Eskes, H., Hovila, J., and Douros, J.: Comparison of TROPOMI/Sentinel-5 Precursor NO2 observations with ground-based measurements in Helsinki, Atmos. Meas. Tech., 13, 205–218, 10.5194/amt-13-205-2020, 2020.Keller, C. A., Long, M. S., Yantosca, R. M., Da Silva, A. M., Pawson, S., and Jacob, D. J.: HEMCO v1.0: a versatile, ESMF-compliant component for calculating emissions in atmospheric models, Geosci. Model Dev., 7, 1409–1417, 10.5194/gmd-7-1409-2014, 2014.Kim, S.-W., Natraj, V., Lee, S., Kwon, H.-A., Park, R., de Gouw, J., Frost, G., Kim, J., Stutz, J., Trainer, M., Tsai, C., and Warneke, C.: Impact of high-resolution a priori profiles on satellite-based formaldehyde retrievals, Atmos. Chem. Phys., 18, 7639–7655, 10.5194/acp-18-7639-2018, 2018.
Krinner, G., Genthon, C., Li, Z.-X., and Le Van, P.: Studies of the Antarctic
climate with a stretched-grid general circulation model, J. Geophys. Res.-Atmos., 102, 13731–13745, 1997.Krol, M., Houweling, S., Bregman, B., van den Broek, M., Segers, A., van Velthoven, P., Peters, W., Dentener, F., and Bergamaschi, P.: The two-way nested global chemistry-transport zoom model TM5: algorithm and applications, Atmos. Chem. Phys., 5, 417–432, 10.5194/acp-5-417-2005, 2005.Li, J., Wang, Y., and Qu, H.: Dependence of Summertime Surface Ozone on
NOx and VOC Emissions Over the United
States: Peak Time and Value, Geophys. Res. Lett., 46,
3540–3550, 10.1029/2018GL081823, 2019.Li, M., Zhang, Q., Kurokawa, J.-I., Woo, J.-H., He, K., Lu, Z., Ohara, T., Song, Y., Streets, D. G., Carmichael, G. R., Cheng, Y., Hong, C., Huo, H., Jiang, X., Kang, S., Liu, F., Su, H., and Zheng, B.: MIX: a mosaic Asian anthropogenic emission inventory under the international collaboration framework of the MICS-Asia and HTAP, Atmos. Chem. Phys., 17, 935–963, 10.5194/acp-17-935-2017, 2017.Li, Y., Henze, D. K., Jack, D., and Kinney, P. L.: The influence of air quality
model resolution on health impact assessment for fine particulate matter and
its components, Air Qual. Atmos. Hlth., 9, 51–68,
10.1007/s11869-015-0321-z, 2016.Li, Y., Pickering, K. E., Barth, M. C., Bela, M. M., Cummings, K. A., and
Allen, D. J.: Evaluation of Parameterized Convective Transport of
Trace Gases in Simulation of Storms Observed During the DC3
Field Campaign, J. Geophys. Res.-Atmos., 123,
11238–11261, 10.1029/2018JD028779, 2018.Lin, H., Feng, X., Fu, T.-M., Tian, H., Ma, Y., Zhang, L., Jacob, D. J., Yantosca, R. M., Sulprizio, M. P., Lundgren, E. W., Zhuang, J., Zhang, Q., Lu, X., Zhang, L., Shen, L., Guo, J., Eastham, S. D., and Keller, C. A.: WRF-GC (v1.0): online coupling of WRF (v3.9.1.1) and GEOS-Chem (v12.2.1) for regional atmospheric chemistry modeling – Part 1: Description of the one-way model, Geosci. Model Dev., 13, 3241–3265, 10.5194/gmd-13-3241-2020, 2020.Long, M. S., Yantosca, R., Nielsen, J. E., Keller, C. A., da Silva, A., Sulprizio, M. P., Pawson, S., and Jacob, D. J.: Development of a grid-independent GEOS-Chem chemical transport model (v9-02) as an atmospheric chemistry module for Earth system models, Geosci. Model Dev., 8, 595–602, 10.5194/gmd-8-595-2015, 2015.Lorente, A., Folkert Boersma, K., Yu, H., Dörner, S., Hilboll, A., Richter, A., Liu, M., Lamsal, L. N., Barkley, M., De Smedt, I., Van Roozendael, M., Wang, Y., Wagner, T., Beirle, S., Lin, J.-T., Krotkov, N., Stammes, P., Wang, P., Eskes, H. J., and Krol, M.: Structural uncertainty in air mass factor calculation for NO2 and HCHO satellite retrievals, Atmos. Meas. Tech., 10, 759–782, 10.5194/amt-10-759-2017, 2017.Mari, C., Jacob, D. J., and Bechtold, P.: Transport and scavenging of soluble
gases in a deep convective cloud, J. Geophys. Res.-Atmos., 105, 22255–22267, 10.1029/2000JD900211, 2000.McDuffie, E., Smith, S., O'Rourke, P., Tibrewal, K., Venkataraman, C., Marais,
E., Zheng, B., Crippa, M., Brauer, M., and Martin, R.: CEDS_GBD-MAPS: Global Anthropogenic Emission Inventory of NOx, SO2, CO, NH3, NMVOCs, BC, and OC from 1970–2017 (2020_v1.0), Zenodo [data set], 10.5281/zenodo.3754964, 2020.
McGregor, J. L. and Dix, M. R.: An updated description of the conformal-cubic
atmospheric model, in: High resolution numerical modelling of the atmosphere
and ocean, Springer, 51–75, 2008.
Miyakoda, K. and Rosati, A.: One-way nested grid models: The interface
conditions and the numerical accuracy, Mon. Weather Rev., 105,
1092–1107, 1977.Murray, L. T., Jacob, D. J., Logan, J. A., Hudman, R. C., and Koshak, W. J.:
Optimized regional and interannual variability of lightning in a global
chemical transport model constrained by LIS/OTD satellite data: IAV
OF LIGHTNING CONSTRAINED BY LIS/OTD, J. Geophys. Res.-Atmos., 117, D20307, 10.1029/2012JD017934, 2012.Palmer, P. I., Jacob, D. J., Chance, K., Martin, R. V., Spurr, R. J. D.,
Kurosu, T. P., Bey, I., Yantosca, R., Fiore, A., and Li, Q.: Air mass factor
formulation for spectroscopic measurements from satellites: Application to
formaldehyde retrievals from the Global Ozone Monitoring Experiment,
J. Geophys. Res.-Atmos., 106, 14539–14550,
10.1029/2000JD900772, 2001.Park, R., Pickering, K., Allen, D., Stenchikov, G., and Fox-Rabinovitz, M.:
Global simulation of tropospheric ozone using the University of Maryland
Chemical Transport Model (UMD-CTM): 2. Regional transport and chemistry over
the central United States using a stretched grid, J. Geophys. Res.-Atmos., 109, D09303, 10.1029/2003JD004269, 2004.Punger, E. M. and West, J. J.: The effect of grid resolution on estimates of
the burden of ozone and fine particulate matter on premature mortality in the
USA, Air Qual. Atmos. Hlth., 6, 563–573,
10.1007/s11869-013-0197-8, 2013.Putman, W. M. and Lin, S.-J.: Finite-volume transport on various cubed-sphere
grids, J. Comput. Phys., 227, 55–78,
10.1016/j.jcp.2007.07.022, 2007.Ramshaw, J. D.: Conservative rezoning algorithm for generalized two-dimensional
meshes, J. Comput. Phys., 59, 193–199,
10.1016/0021-9991(85)90141-X, 1985.Rienecker, M. M., Suarez, M. J., Todling, R., Bacmeister, J., Takacs, L., Liu, H. C., Gu, W., Sienkiewicz, M., Koster, R. D., Gelaro, R., Stajner, I., and Nielsen, J. E.: The GEOS-5 Data Assimilation System: Documentation of Versions 5.0.1, 5.1.0, and 5.2.0., available at https://gmao.gsfc.nasa.gov/pubs/docs/Rienecker369.pdf (last access: 15 April 2021), 2008.
Schmidt, F.: Variable fine mesh in spectral global models, Beitr. Phys. Atmos.,
50, 211–217, 1977.
Slingo, J., Bates, K., Nikiforakis, N., Piggott, M., Roberts, M., Shaffrey, L.,
Stevens, I., Vidale, P. L., and Weller, H.: Developing the next-generation
climate system models: challenges and achievements, Philos.
T. R. Soc. A-Math., 367, 815–831, 2009.
Srivastava, R. K., McRae, D., and Odman, M.: An adaptive grid algorithm for
air-quality modeling, J. Comput. Phys., 165, 437–472, 2000.Stettler, M., Eastham, S., and Barrett, S.: Air quality and public health
impacts of UK airports. Part I: Emissions, Atmos. Environ.,
45, 5415–5424, 10.1016/j.atmosenv.2011.07.012, 2011.Suarez, M., Trayanov, A., Hill, C., Schopf, P., and Vikhliaev, Y.: MAPL: a
high-level programming paradigm to support more rapid and robust encoding of
hierarchical trees of interacting high-performance components, in:
Proceedings of the 2007 symposium on Component and framework technology in
high-performance and scientific computing – CompFrame '07, ACM
Press, Montreal, Quebec, Canada, p. 11, 10.1145/1297385.1297388, 2007.Tack, F., Merlaud, A., Iordache, M.-D., Pinardi, G., Dimitropoulou, E., Eskes, H., Bomans, B., Veefkind, P., and Van Roozendael, M.: Assessment of the TROPOMI tropospheric NO2 product based on airborne APEX observations, Atmos. Meas. Tech., 14, 615–646, 10.5194/amt-14-615-2021, 2021.The International GEOS-Chem User Community:geoschem/geos-chem: GEOS-Chem 12.5.0 (12.5.0), Zenodo [code], 10.5281/zenodo.3403111, 2019.The International GEOS-Chem User Community: geoschem/gchp: GCHP 13.0.0-alpha.3 (13.0.0-alpha.3), Zenodo [code], 10.5281/zenodo.4317978, 2020.Thompson, M., Auer, B., Clune, T., Trayanov, A., Jiang, W., Norris, P.,
Lundgren, L., Sherman, E., Jamieson, W., Bindle, L., Eastham, S. D., Putman,
B., and Vikhliaev, Y.: GEOS-ESM/MAPL: MAPL 2.2 Release (v2.2.0), Zenodo [code], 10.5281/zenodo.3903435, 2020.Todling, R. and El Akkraoui, A.: The GMAO Hybrid Ensemble-Variational
Atmospheric Data Assimilation System: Version 2.0, available at https://gmao.gsfc.nasa.gov/pubs/docs/Todling1019.pdf (last access: 15 April 2021), 2018.
Tomita, H.: A stretched icosahedral grid by a new grid transformation, J. Meteorol. Soc. Jpn. Ser. II, 86, 107–119, 2008.
Tomlin, A., Berzins, M., Ware, J., Smith, J., and Pilling, M.: On the use of
adaptive gridding methods for modelling chemical transport from multi-scale
sources, Atmos. Environ., 31, 2945–2959, 1997.
Trieu, T. T. N., Goto, D., Yashiro, H., Murata, R., Sudo, K., Tomita, H.,
Satoh, M., and Nakajima, T.: Evaluation of summertime surface ozone in Kanto
area of Japan using a semi-regional model and observation, Atmos.
Environ., 153, 163–181, 2017.
Uchida, J., Mori, M., Nakamura, H., Satoh, M., Suzuki, K., and Nakajima, T.:
Error and energy budget analysis of a nonhydrostatic stretched-grid global
atmospheric model, Mon. Weather Rev., 144, 1423–1447, 2016.Valin, L. C., Russell, A. R., Hudman, R. C., and Cohen, R. C.: Effects of model resolution on the interpretation of satellite NO2 observations, Atmos. Chem. Phys., 11, 11647–11655, 10.5194/acp-11-11647-2011, 2011.van der Werf, G. R., Randerson, J. T., Giglio, L., van Leeuwen, T. T., Chen, Y., Rogers, B. M., Mu, M., van Marle, M. J. E., Morton, D. C., Collatz, G. J., Yokelson, R. J., and Kasibhatla, P. S.: Global fire emissions estimates during 1997–2016, Earth Syst. Sci. Data, 9, 697–720, 10.5194/essd-9-697-2017, 2017.Wang, Y. X., McElroy, M. B., Jacob, D. J., and Yantosca, R. M.: A nested grid
formulation for chemical transport over Asia: Applications to CO:
Nested grid co simulation over Asia, J. Geophys. Res.-Atmos., 109, D22307, 10.1029/2004JD005237, 2004.Weng, H., Lin, J., Martin, R., Millet, D. B., Jaeglé, L., Ridley, D., Keller,
C., Li, C., Du, M., and Meng, J.: Global high-resolution emissions of soil
NOx, sea salt aerosols, and biogenic volatile organic compounds, Sci.
Data, 7, 148, 10.1038/s41597-020-0488-5, 2020.Wild, O. and Prather, M. J.: Global tropospheric ozone modeling: Quantifying
errors due to grid resolution, J. Geophys. Res., 111,
D11305, 10.1029/2005JD006605, 2006.Yan, Y., Lin, J., Chen, J., and Hu, L.: Improved simulation of tropospheric ozone by a global-multi-regional two-way coupling model system, Atmos. Chem. Phys., 16, 2381–2400, 10.5194/acp-16-2381-2016, 2016.Yan, Y.-Y., Lin, J.-T., Kuang, Y., Yang, D., and Zhang, L.: Tropospheric carbon monoxide over the Pacific during HIPPO: two-way coupled simulation of GEOS-Chem and its multiple nested models, Atmos. Chem. Phys., 14, 12649–12663, 10.5194/acp-14-12649-2014, 2014.
Zängl, G., Reinert, D., Rípodas, P., and Baldauf, M.: The ICON
(ICOsahedral Non-hydrostatic) modelling framework of DWD and MPI-M:
Description of the non-hydrostatic dynamical core, Q. J.
Roy. Meteor. Soc., 141, 563–579, 2015.
Zhang, D.-L., Chang, H.-R., Seaman, N. L., Warner, T. T., and Fritsch, J. M.: A
two-way interactive nesting procedure with variable terrain resolution,
Mon. Weather Rev., 114, 1330–1339, 1986.Zhao, X., Griffin, D., Fioletov, V., McLinden, C., Cede, A., Tiefengraber, M., Müller, M., Bognar, K., Strong, K., Boersma, F., Eskes, H., Davies, J., Ogyu, A., and Lee, S. C.: Assessment of the quality of TROPOMI high-spatial-resolution NO2 data products in the Greater Toronto Area, Atmos. Meas. Tech., 13, 2131–2159, 10.5194/amt-13-2131-2020, 2020.Zhuang, J., Jacob, D. J., Lin, H., Lundgren, E. W., Yantosca, R. M., Gaya,
J. F., Sulprizio, M. P., and Eastham, S. D.: Enabling High‐Performance
Cloud Computing for Earth Science Modeling on Over a Thousand
Cores: Application to the GEOS‐Chem Atmospheric Chemistry
Model, J. Adv. Model. Earth Sy., 12, e2020MS002064,
10.1029/2020MS002064, 2020.