Grid-Stretching Capability for the GEOS-Chem 13.0.0 Atmospheric Chemistry Model

Modeling atmospheric chemistry at fine resolution globally is computationally expensive; the capability to focus on specific geographic regions using a multiscale grid is desirable. Here, we develop, validate, and demonstrate stretched-grids in the GEOS-Chem atmospheric chemistry model in its high performance implementation (GCHP). These multiscale grids are specified at runtime by four parameters that offer users nimble control of the region that is refined and the resolution of the refinement. We validate the stretched-grid simulation versus global cubed-sphere simulations. We demonstrate the operation 5 and flexibility of stretched-grid simulations with two case studies that compare simulated tropospheric NO2 column densities from stretched-grid and cubed-sphere simulations to retrieved column densities from the TROPOspheric Monitoring Instrument (TROPOMI). The first case study uses a stretched-grid with a broad refinement covering the contiguous US to produce simulated columns that perform similarly to a C180 (∼50 km) cubed-sphere simulation at less than one-ninth the computational expense. The second case study experiments with a large stretch-factor for a global stretched-grid simulation with 10 a highly localized refinement with ∼10 km resolution for California. We find that the refinement improves spatial agreement with TROPOMI columns compared to a C90 cubed-sphere simulation of comparable computational demands, despite conducting the simulation at a finer resolution than parent meteorological fields. Overall we find that stretched-grids in GEOS-Chem are a practical tool for fine resolution regionalor continental-scale simulations of atmospheric chemistry. Stretched-grids are available in GEOS-Chem version 13.0.0. 15 1 https://doi.org/10.5194/gmd-2020-398 Preprint. Discussion started: 16 December 2020 c © Author(s) 2020. CC BY 4.0 License.

and flexibility of stretched-grid simulations with two case studies that compare simulated tropospheric NO 2 column densities from stretched-grid and cubed-sphere simulations to retrieved column densities from the TROPOspheric Monitoring Instrument (TROPOMI). The first case study uses a stretched-grid with a broad refinement covering the contiguous US to produce simulated columns that perform similarly to a C180 (∼50 km) cubed-sphere simulation at less than one-ninth the computational expense. The second case study experiments with a large stretch-factor for a global stretched-grid simulation with 10 a highly localized refinement with ∼10 km resolution for California. We find that the refinement improves spatial agreement with TROPOMI columns compared to a C90 cubed-sphere simulation of comparable computational demands, despite conducting the simulation at a finer resolution than parent meteorological fields. Overall we find that stretched-grids in GEOS-Chem are a practical tool for fine resolution regional-or continental-scale simulations of atmospheric chemistry. Stretched-grids are available in GEOS-Chem version 13.0.0.

Introduction
Global simulations of atmospheric chemistry are computationally demanding. Chemical mechanisms in the troposphere typically involve more than 100 chemical species, emitted by anthropogenic and natural sources, mixing through 3-D transport on all scales. Typical global model resolutions are on the order of hundreds of kilometers and generally limited by the availability of computational resources. Massively parallel models such as GEOS-Chem in its high-performance implementation (GCHP; 20 Eastham et al. (2018)) can run on more than 1000 cores (Zhuang et al., 2020) with demonstrated capability of 50 km resolution.
The coarse resolution of global models can lead to systematic errors in applications when scales of variability finer than the model resolution are relevant, such as vertical transport and scavenging by convective updrafts (Mari et al., 2000;Li et al., 2018Li et al., , 2019, nonlinear chemistry such as NO x titration (Valin et al., 2011), localized emission sources (Davis et al., 2001;Freitas et al., 2007), a priori profiles for satellite retrievals (Heckel et al., 2011;Goldberg et al., 2017;Kim et al., 2018), and 25 simulated concentrations for population exposure estimates (Punger and West, 2013;Li et al., 2016). Nested grids are commonly used for simulations that need to capture fine-scale modes of variability. With one-way nested grids, a global simulation generates boundary conditions for a fine resolution regional simulation (Wang et al., 2004;. With two-way nested grids, the global simulation is dynamically coupled to the regional simulation (Yan et al., 2014;Feng et al., 2020). An alternative type of multiscale grids that are well-established in the regional climate modeling community are stretched-grids 30 (Fox-Rabinovitz et al., 2006, 2008. Stretched-grids are deformed model grids with increased grid density in the region of interest, and transition smoothly between the refinement and coarser global resolutions. Stretched-grids have the advantage of being inherently two-way coupled, and since stretching does not change the grid topology, grid-independent models like GEOS-Chem can implement stretched-grids without structural changes or online simulation couplers. Stretched-grid simulations have little overhead and are economical because their computational cost is similar to a cubed-sphere simulation with 35 the same grid size. Here we implement, validate, and demonstrate stretched-grids in GEOS-Chem, enabling massively parallel global multiscale simulations of atmospheric chemistry. Several recent works set the stage for the development of grid-stretching in GEOS-Chem. Long et al. (2015) developed the grid-independent capability of GEOS-Chem. Harris et al. (2016) developed the capability for stretched-grids in the GFDL Finite-Volume Cubed-Sphere Dynamical Core (FV3), which GCHP uses to calculate advection. Eastham et al. (2018) de-40 veloped the capability for GEOS-Chem to operate on cubed-sphere grids in a distributed memory framework for massive parallelization, and to use the Model Analysis and Prediction Layer (MAPL) of the NASA Global Modeling and Assimilation Office (GMAO) together with the Earth System Modeling Framework (ESMF) to couple model components. GEOS-Chem version 12.5.0 added grid-independent emissions that produce consistent emissions regardless of the model grid (Weng et al., 2020;The International GEOS-Chem User Community, 2019). Most recently, MAPL version 2 (Thompson et al., 2020) of the 45 NASA GMAO added stretched-grid support.
This manuscript describes the development and validation of the grid-stretching capability in GCHP and discusses practical considerations for running stretched-grid simulations. Sect. 2.1 provides an overview of GCHP and its underlying gnomonic cubed-sphere grid. Sect. 2.2 describes stretching the grid with the Schmidt (1977) transform following the methodology of appropriate stretch-factor. Sect. 2.4 describes the model configuration for the simulations in the manuscript. Sect 2.5 summarizes the testing of stretched-grids in GCHP and provides a comparison of simulated oxidants and PM 2.5 concentrations from stretched-grid and cubed-sphere simulations. Sect. 3 presents two case studies that demonstrate and explore stretchedgrid applications, with comparisons of stretched-grid and cubed-sphere simulations with observations from TROPOspheric Monitoring Instrument (TROPOMI). We use GEOS-Chem version 13.0.0 in its high-performance implementation (GCHP; Eastham et al. (2018)). GEOS-Chem, originally described in Bey et al. (2001), simulates tropospheric-stratospheric chemistry by solving 3-D chemical continuity equations. GCHP uses MAPL (Suarez et al., 2007) and ESMF (Hill et al., 2004), which facilitate the coupling of model 60 components and the use of the High-Performance Computing (HPC) infrastructure. The 3-D advection component is the GFDL Finite-Volume Cubed-Sphere Dynamical Core (Putman and Lin, 2007). Columnar operators (Long et al., 2015) are used for columnar or local calculations such as convection and chemical kinetics. Emissions are aggregated, parameterized, and computed with the Harmonized Emissions Component (HEMCO) described in Keller et al. (2014). Offline meteorological data is from the Goddard Earth Observing System (GEOS) data assimilation system. All regridding, including the regridding of emissions data and meteorological data, is performed online by ESMF. GCHP discretizes the atmosphere with a gnomonic cubed-sphere grid with levels extending from the surface to 1 Pa. The cubed-sphere grid has several advantages over the conventional latitude-longitude grid, stemming from its more uniform grid-boxes that benefit the parallelization and numerical stability of transport (Eastham et al., 2018). The horizontal resolution of a GCHP simulation is a key determinant of its computational demands.

Grid-stretching
The grid-stretching procedure in GCHP, described here, uses a simplified form of the Schmidt transform for gnomonic cubedsphere grids, and follows the methodology of Harris et al. (2016). The Schmidt (1977) transform can be applied to any grid, and effectively stretches the grid to increase its density in a region. The grid-stretching procedure has two steps and starts with a gnomonic cubed-sphere grid. First, the grid is refined at the South Pole by remapping the grid coordinate latitudes with a 75 modified Schmidt transform where φ is an input latitude, φ is the output latitude, and S is a parameter called the stretch-factor. The stretch-factor controls the strength of this remapping operation, which effectively attracts the grid coordinates towards the South Pole along meridians.
The second step is rotating the entire grid so that the refinement at the South Pole is repositioned to the region of interest. The user specifies a target latitude and target longitude (φ t , θ t ), and the refinement is re-centered to this coordinate. In GCHP, according to the right-hand-rule, these rotations are φ t + 90 about 90 • E and θ t + 180 about 90 • N. Note that nonstretched cubed-sphere grids in GCHP have a −10 • E offset, so a stretched-grid with parameters S = 1, φ t = −90 • N, θ t = 170 • E is identical to a nonstretched cubed-sphere grid. Figure 1 illustrates the effect of S on stretching a cubed-sphere grid. A stretch-factor greater than one causes stretching.

85
Larger stretch-factors cause more stretching, and result in a finer and more localized refinement. The resolution at the center of the refinement is approximately S times finer than it was before stretching, and similarly, the antipode resolution is approximately S times coarser. These relative changes are approximate since the Schmidt transform is continuous and the grid-boxes have nonzero length edges. The grids in Figure 1 illustrate three noteworthy features of stretched-grids: (1) the changes in resolution are smooth, (2) the refined domain gets smaller as S increases, and (3) grid-boxes outside the refined domain expand.

90
The cubed-sphere face at the center of the refined domain is called the target-face.
The relative change to a grid-box's size from stretching can be quantified by local scaling. This quantity represents the effect of grid-stretching at a given point. For a stretch-factor of S, the local scaling at a given point depends exclusively on how far that point is from the target coordinate. Local scaling, L, can be derived from Eqn. 1, and expressed as where Θ is the angular distance to the target point. Appendix B contains the derivation of Eqn. 2. Figure 2 shows local scaling as a function of distance, for stretch-factors between 1 and 10. Overlaid are dashed lines that show the distance of cubed-sphere face edges after stretching. In the target-face, local scaling is approximately 1/S and nearly constant. Grid-stretching refines the resolution to the distance where L = 1, and coarsens the resolution at farther distances. Four scalar parameters fully describe a stretched-grid: the size of the cubed-sphere, the stretch-factor (S), the target latitude 100 (φ t ), and the target longitude (θ t ). These concise parameters are conceptually simple, precise, and give the user nimble control of the grid. The combination of cubed-sphere size and stretch-factor controls the grid resolution. The target latitude and longitude specify the center of the refined domain. Moderate stretch-factors (e.g., 1.4-3.0) are suitable for broad refinements for continental-scale studies. Large stretch-factors (e.g., >5.0) are suitable for localized refinements for regional-scale studies.

105
Although the stretch-factor is a well-defined parameter, an appropriate value is application-specific and moderately variable.
For computational efficiency, it is desirable to use the largest viable stretch-factor, to achieve the finest refinement for a given cubed-sphere size. However, larger stretch-factors also result in a smaller domain being refined and a coarser resolution at the target's antipode. To determine the maximum suitable stretch-factor for a given application, one should consider the size of the domain to be refined, and the sensitivity of the study species to coarse resolution outside the refined domain. It is also 110 worth noting that there is limited prior model evaluation of GEOS-Chem at resolutions coarser than C24 (∼400 km). A simple procedure for choosing a stretch-factor, S, is choosing the maximum stretch-factor subject to two constraints: 1. Constraining S by the size of the refined domain. Local scaling is approximately constant and equal to 1/S throughout the target-face. Therefore, defining the refined domain as the region whose resolution is enhanced by a factor of ∼ S, the target-face is a reasonable approximation of the refined domain. A constraint for S, such that the width of the target-face 115 is greater than w tf , is S < 0.414 cot (w tf /4 r E ) where r E is Earth's radius. For example, for a refined domain with a diameter of at least 3000 km, S < 3.5. Alternatively, one can inspect Figure 2 to find the distance from the target point to the edge of the target-face for a given value of S.

2.
Constraining S by a maximum and minimum resolution. The target resolution is S 2 times finer than the antipodal resolution; therefore, a constraint for S based on the desired maximum resolution, R max , and minimum resolution, R min , 120 is S < R max /R min . For example, for a maximum and minimum resolution comparable to a nonstretched C360 and C24 cubed-sphere, S < 3.9.
Once the constraints for S are determined, one can choose S and the grid size for their simulation. It is worth noting that GCHP requires that the grid size is an even integer (e.g., C88, C90, and C92). For example, for a stretched-grid with refined domain with a diameter greater than 3000 km, a maximum resolution of C360, and a minimum resolution of C24, the constraints would 125 be S < 3.5 and S < 3.9. Then, one would choose a cubed-sphere size of C104 and S = 3.46.

Model configuration
All simulations in this manuscript use GCHP version 13.0.0-alpha.3 with a common configuration. Table 1  with a suffix denoting the region that is refined. For example, a stretched-grid simulation with an effective resolution of C180 in the contiguous US is named "C180e-US". "C180e" refers to the stretched-grid refinement being comparable to the resolution of a C180 cubed-sphere grid.

Stretched-grid development and validation
Developing and testing stretched-grids in GCHP involved multiple stages. The first step was developing a prototype simulation 145 to confirm the functionality of grid-stretching in each model component. The prototype simulation excluded chemistry and used a simplified set of emissions. Once the prototype simulation was operational, full chemsitry and the full set of emissions were enabled. Specialized benchmarking code was then used to compare stretched-grid and cubed-sphere simulations, to debug and test the implementation.
Next we compare the concentrations of oxidants and PM 2.5 from cubed-sphere and stretched-grid simulations. The domain 150 for this comparison is the contiguous US, and a C96 cubed-sphere simulation was chosen as the control (C96-global). The stretched-grid simulation, C96e-NA, had a grid size of C48, and its parameters were chosen so its average resolution was the same as C96-global in the contiguous US. C96e-NA's parameters were S = 2.4, φ t = 35 • N, and θ t = 96 • W. Figure 3 compares the resolution of C96e-NA and C96-global grids. Note that the stretch-factor is 2.4, rather than 2.0, because the cubed-sphere's quasi-uniform resolution is finer in the US than its nominal global resolution. A consequence of C96e-NA and

Stretched-Grid Case-Studies
Next, we focus on two case studies to further demonstrate the operation of stretched-grid simulations and the flexibility of stretched-grid refinements. The first is a typical case that considers a stretched-grid simulation with a moderate stretch-factor for a broad refinement covering the contiguous US at ∼50 km (C180e) resolution. The second is an exploratory case that 170 experiments with a large stretch-factor for a localized refinement covering California at ∼10 km (C900e) resolution. The resolution of our second case study (∼10 km) is more than twice as fine as GEOS-Chem's native resolution of C360 (∼25 km) determined by the GEOS assimilated meteorological data.
We focus on NO 2 as a well measured species that is sensitive to resolution. Simulated tropospheric NO 2 column densities from stretched-grid and cubed-sphere simulations are compared to retrieved column densities from TROPOMI (Veefkind  densities at 3.5×5.5 km 2 resolution use a modified version of the Dutch OMI NO 2 (DOMINO) retrieval algorithm (Boersma et al., 2011(Boersma et al., , 2018. We include observations with retrieved cloud fractions less than 10 %. Retrieved NO 2 column densities are sensitive to the a priori profiles used to calculate the air mass factors (Boersma et al., 2018;Lorente et al., 2017). To 180 avoid spurious differences from the retrieval's a priori profiles when comparing simulated and retrieved NO 2 column densities, we recalculate the air mass factors with the mean simulated relative vertical profiles (shape factors) from the stretched-grid simulations following the approach described in Cooper et al. (2020) and Palmer et al. (2001). Evaluations of TROPOMI NO 2 columns show good correlation with ground-based measurements with a small low bias (Griffin et al., 2019;Ialongo et al., 2020;Zhao et al., 2020;Tack et al., 2020). The computational demands of C180e-US were significantly less than C180-global and comparable to C60-global. Table 2 gives timing test results for C180-global, C180e-US, and C60-global. The total CPU time for C180e-US was nearly a factor of 20 less than C180-global, resulting from fewer grid-boxes and reduced overhead. The total CPU time for C180e-US was 15 % 205 greater than C60-global. The slight increase of data input cost in C180e-US is suspected to be caused by a load imbalance in online input regridding in ESMF. Work to mitigate this load imbalance is underway.

A stretched-grid simulation with a large stretch-factor
Stretched-grids with large stretch-factors have localized refinements. Here we experiment with a large stretch-factor to create a localized refinement covering California at ∼10 km resolution. Our focus on California is motivated by the pronounced heterogeneity in sources and topography that has challenged global models. The stretched-grid simulation, C900e-CA, has a grid size of C90 and stretch parameters S = 10, φ t = 37.2 • N, and θ t = 119.5 • W. This simulation demonstrates that computationally, stretched-grid simulations are capable of very fine resolutions that are comparable to those of regional models. This experiment leverages the fine resolution of anthropogenic NO emissions from the NEI-2011 inventory (0.1 • ×0.1 • or ∼9 km).
The meteorological data is conservatively downscaled from its native resolution of 0.25 • ×0.3125 • . This downscaling, along 215 with all other regridding is performed online by ESMF. Work is underway in the GCHP-GMAO community to prepare finer resolution meteorological inputs for GEOS-Chem (including more vertical levels). Figure 7 shows the grids of C900e-CA and a nonstretched C90 cubed-sphere simulation, C90-global. The average resolution of C900e-CA in California is 11.2 km. The large stretch-factor used in C900e-CA causes some grid-boxes to expand significantly. Local scaling is useful for understanding the variability of the grid's resolution. For example, New York is approximately 4000 km from C900e-CA's target point. Figure   220 2 shows that for S = 10, the local scaling at 4000 km is close to 1. Equivalently, substituting S = 10 and Θ = 4000 km/r E 11 https://doi.org/10.5194/gmd-2020-398 Preprint. Discussion started: 16 December 2020 c Author(s) 2020. CC BY 4.0 License.   in Eqn. 2 gives L = 1.04. Therefore, the resolution of C900e-CA in New York is similar to a nonstretched C90 cubed-sphere.
This can be confirmed in Figure 7.   Stretched-grids enable multiscale grids in GCHP and complement the other multiscale grid methods that are available in GEOS-Chem variants. The primary benefit of grid-stretching is ease of use. The refinement is flexible and controlled by four simple runtime parameters. Stretched-grids operate naturally in GCHP, so switching between stretched-grids and cubed-sphere grids is seamless. Stretched-grid simulations are standalone simulations, and do not require any pregenerated or dynamically-refinement, and that one cannot control the refined domain, refinement resolution, and global resolution independently.
Stretched-grid simulations can be used for regional-or continental-scale simulation purposes. Generally, stretch-factors in the range 1.4-4.0 are applicable for large refined domains. Higher stretch-factors can be used for very fine resolution simulations for regional-scale applications. To aid in choosing an appropriate stretch-factor, we propose a simple procedure based on choosing the maximum stretch-factor subject to two constraints: the size of the refinement, and the maximum and 15 https://doi.org/10.5194/gmd-2020-398 Preprint. Discussion started: 16 December 2020 c Author(s) 2020. CC BY 4.0 License. Figure A1. Variability of a C180 cubed-sphere's resolution.
Appendix A: Variability of cubed-sphere grid resolution Figure A1 shows the variability of a C180 cubed-sphere grid's resolution. The coarsest resolutions is at the center of faces, and 260 the finest resolutions are at the corners of the faces. The average resolution is 51.1 km. The resolution at the center of the face is 61.9 km (21.1 % greater than the average resolution). The resolution at the corner of the face is 40.6 km (25.8 % less than the average resolution).

Appendix B: Derivation of local scaling
Local scaling, L, is the relative change to a grid-box's length from grid-stretching. Consider a line segment that follows a S 2 (1 − sin y) + sin y + 1 (B3) We can obtain the local scaling at y, rather than y , by substituting the inverse of the Schmidt transform into Eqn. B3. This gives 275 L(y; S) = S 2 (1 + sin y) − sin y + 1 2S (B4) Finally, we can generalize Eqn. B5 so it is a function of arclength from the target. The center of the refinement after applying Eqn. 1 is at the South Pole. The arclength from y to the South Pole is Θ = y − (−π/2). Substituting y = Θ − π/2 into Eqn. B5 gives L(Θ; S) = S 2 (1 − cos Θ) + cos Θ + 1 2S (B5)