Atmospheric chemistry models are used to address a wide range of problems related to climate forcing, air quality, and atmospheric deposition. Simulations of oxidant and aerosol chemistry involve hundreds of chemically interacting species, coupled to transport on all scales. The computational demands are considerable, which has limited the inclusion of atmospheric chemistry in climate models (National Research Council, 2012). Offline chemical transport models (CTMs), where meteorology is provided as input data from a parent global climate model (GCM) or atmospheric data assimilation system (DAS), are frequently used for reasons of simplicity, reproducibility, and ability to focus on chemical processes. The global GEOS-Chem CTM originally described by Bey et al. (2001), using meteorological input from the Goddard Earth Observation System (GEOS) DAS of the NASA Global Modeling and Assimilation Office (GMAO), is used by hundreds of atmospheric chemistry research groups worldwide (http://www.geos-chem.org, last access: 19 July 2018). Increasing computational resources in the form of massively parallel architectures can allow GEOS-Chem users to explore more complex problems at higher grid resolutions, but this requires re-engineering of the model to take advantage of these architectures. Here, we describe a high-performance version of GEOS-Chem (GCHP) engineered for this purpose, and we demonstrate its ability to access a new range of capability and scales for global atmospheric chemistry modeling.

The original GEOS-Chem CTM (GEOS-Chem Classic, or GCC) was designed for shared-memory (OpenMP) parallelization. A detailed description of the model including a user manual is available on the GEOS-Chem website (http://www.geos-chem.org, last access: 19 July 2018). Computation is distributed over a number of cores on a single node, with data held in shared arrays. But recent growth in computational power has taken the form of massively parallel networked systems, where additional computational power is achieved by increasing the number of identical nodes rather than by improving the nodes themselves. This has placed a restriction on growth in the problem size and complexity which can be solved by a single instance of GCC. To take advantage of massively parallel architectures, a new framework is needed which allows GEOS-Chem to use a distributed-memory model, where the computation is distributed across multiple coordinated nodes using a Message Passing Interface (MPI) implementation such as MVAPICH2 or OpenMPI.

An important first step in this evolution was the integration of GEOS-Chem as the online chemistry component within the GEOS DAS (Long et al., 2015). In order to ensure that the online and offline versions of GEOS-Chem were identical, GCC was modified to use the exact same code in the independent CTM and in the DAS. Major modifications were required to make GEOS-Chem grid-independent and compatible with the GEOS Modeling and Analysis Prediction Layer (MAPL) (Suarez et al., 2007), an Earth System Modeling Framework (ESMF) (Hill et al., 2004) based software layer which handles communication between different components of the GEOS DAS. The GEOS-Chem code was adapted to accept an arbitrarily sized horizontal set of atmospheric columns, with no requirements regarding adjacency of the columns or overall coverage of any particular set. All these changes were made “under the hood” in the standard GEOS-Chem code. When GEOS-Chem is run as GCC, the set of columns is designated as a single block which covers the entire globe or a subset in a nested domain, and parallelization is achieved by internally running parallel loops over the columns. When GEOS-Chem is run as part of GEOS, MAPL internally splits the atmosphere into smaller domains, each of which contains a different set of atmospheric columns. These domains can then be distributed across multiple nodes, exploiting massively parallel architectures. As a result of these changes, the same GEOS-Chem code can now be run either as a stand-alone, shared-memory offline CTM, or as a GCM component in the massively parallel, distributed GEOS DAS. Any improvement in chemical modeling developed for the offline CTM is thus immediately available in the GEOS DAS version, which never becomes out of date and remains referenceable to the current version of GEOS-Chem.

In this work, we take the next logical step of developing GCHP as a distributed-memory, MAPL-based implementation of the GEOS-Chem CTM. GCHP uses an identical copy of the GEOS-Chem Classic (GCC) code to provide the same high-fidelity atmospheric chemical simulation capabilities, allowing users to switch between GCC and GCHP implementations with confidence that they are using the same model. The exact same internal code is used in GCC shared-memory and GCHP distributed-memory applications. This closes the development loop between online and offline modeling. By sharing infrastructure code between GCHP and GEOS in the form of MAPL, offline modelers can now take advantage of modeling advances which originate in the online model in the same way that GEOS benefits from advances in chemical modeling developed in the GEOS-Chem CTM (Nielsen et al., 2017). By way of example, GEOS was recently able to conduct a full-year 13 km resolution “nature run” with the current standard version of GEOS-Chem tropospheric chemistry (Hu et al., 2018). In return, GCHP incorporates the more efficient cubed-sphere grid and the Finite-Volume Cubed-Sphere Dynamical Core (FV3) advection code present in GEOS, and is capable of directly ingesting GEOS output in its native cubed-sphere format.

We analyzed the performance of GCHP v11-02c by conducting simulations at multiple grid resolutions (C24 to C180), each for a range of core counts (Table 1). For low-resolution applications, performance is also compared to the maximum achievable performance using the GCC v11-02c shared-memory architecture. All simulations are for 1 month (July 2016) of troposphere–stratosphere oxidant–aerosol chemistry, including 206 species and 135 tracers, and using operational meteorological data from GEOS forward processing (FP). The GCC v11-02c simulations use previously regridded 2∘×2.5∘ and 4∘×5∘ meteorological fields, while the GCHP v11-02c simulations regrid the 0.25∘×0.3125∘ fields to the cubed sphere on the fly through ExtData. A native-resolution cubed-sphere GEOS data output stream is presently under development at GMAO and will benefit GCHP by reducing the need for regridding.

All simulations were conducted on the Harvard Odyssey computational cluster. Both GCHP and GCC were compiled using the Intel Fortran compiler (v15.0.0), and MPI capabilities for GCHP were provided by OpenMPI (v1.10.3). Each node of the cluster has 32 Intel Broadwell 2.1 GHz cores sharing 128 GB of RAM, and all nodes are connected via Mellanox FDR Infiniband. Input and output data are stored using a Lustre parallel file system, accessible through the same Infiniband network fabric. All simulations were scheduled to enforce exclusive access to the nodes, preventing possible performance degradation due to sharing of node resources.

Figure 3 shows the total time taken to perform the simulation at each resolution, both in terms of wall time and in terms of the time per 1000 simulated atmospheric columns in the model grid. Results using the conventional GCC shared-memory platform at 4∘×5∘ and 2∘×2.5∘ resolutions are also shown for comparison.

At the lowest simulated resolution (C24), GCHP's runtime exceeds that of GEOS-Chem Classic. This is predominantly due to overhead associated with file open operations, as the native-resolution meteorological data used to drive GCHP are opened and read independently for each field. This effect is clearly visible in the lower plot of Fig. 3, where performance penalties due to this overhead result in significantly longer runtimes per 1000 columns at C24 compared to C180. This can be addressed in the future through both structural changes and parallelization of the input.

After doubling the resolution to C48 (2∘×2.5∘), GCHP begins to outperform GCC, with a reduced overall simulation time even at core counts which are currently accessible to GCC, despite the larger input requirements of GCHP. A 1-month simulation at C48 requires only 6 h using 96 cores. GCHP scalability also improves as the model resolution increases. At C180, the reduction in simulation time for each doubling in the number of cores is approximately a factor of 1.6.

Two factors affect GCHP scalability: fixed costs and overhead. Fixed costs are for operations which run on a fixed number of cores, regardless of the number of cores dedicated to the simulation. Overhead is the need for additional coordination and data communication between cores, which grows with the number of cores. Eventually, the overhead of the additional cores exceeds the computational benefit, resulting in a performance plateau. This overhead includes one-off costs, such as the MPI interface initialization, which can be significant when running with a large number of cores but which can be reduced in relative terms by running longer simulations.

The scalability of each model component is shown in Fig. 4. This shows the total time spent on each component at C48 and C180 resolution as a function of the total number of cores used, from 6 cores up to 540. We see that the dominant fixed cost at both C48 and C180 is input, which also dominates the overall cost for C48 with more than 48 cores. This is due to the serial nature of the current input code, overhead associated with file open operations, and the aforementioned use of native-resolution meteorological data for even low-resolution GCHP simulations. Output operations are a second fixed cost, being handled by a single core at all times. For these simulations, 22 3-D fields were stored with hourly frequency, and output was a minor contributor to overall costs. Fixed costs can be converted into scalable costs by parallelizing the component in question, and this is a future work agenda.

Chemistry, advection, and convection all scale well with increasing core count. Chemistry is the most expensive process at both coarse (C48) and fine (C180) resolution but has near-perfect scalability. Thus, at C48, we see that input becomes the limiting process when the number of cores exceeds 48. Advection and other processes show more departure from perfect scalability, and may dominate the time requirement as the number of cores exceeds 600. The scalability of advection suffers from the additional communication overhead associated with reducing the domain size, as each domain must communicate a larger proportion of its concentration data to its neighbors. More time is spent on communication relative to computation. However, wall time for advection does consistently fall with increasing core counts, an improvement compared to Long et al. (2015), where wall time increased as core counts exceeded 200 for a grid resolution equivalent to C48. We attribute this to the change from a latitude–longitude grid to the more scalable cubed-sphere grid.

The remaining wall time is taken up by the “other” component, a mix of scalable and non-scalable processes. This includes the one-off cost of initializing the MPI interface, which grows non-linearly with the number of cores. At C180, these costs are still exceeded by scalable costs when running with 540 cores, so no plateau in performance is observed.

The primary advantage of GCHP is the ability to perform simulations of atmospheric chemistry at resolutions previously not available to the community. Figure 5 shows illustrative distributions of simulated ozone concentration at 4 km altitude, simulated at C24 (≈4∘×5∘) and C180 (≈0.5∘×0.625∘). Simulations were performed using 24 and 360 cores, respectively. Results are also shown from simulations using GCC, at 4∘×5∘ and 2∘×2.5∘. Emissions are identical for all simulations.

Global-scale patterns in ozone concentration are not substantially affected by the increase in resolution, as these are determined by large-scale processes. The agreement between the two simulations illustrates the consistency of GCHP across scales. However, increasing the horizontal resolution improves the ability of the model to capture the behavior of intercontinental plumes (Eastham and Jacob, 2017). The consequences of this are visible in the ozone distributions over the Pacific and Atlantic. We also observe maxima in the coarse-resolution simulation which are not visible in the finer-resolution simulation, such as the peak in ozone concentration over Egypt. This suggests possible simulation biases at coarse resolution.

Increasing the horizontal resolution also improves the ability of the model to resolve features at the scale of local air quality, as demonstrated by the plots of daily average aerosol optical depth (AOD) shown in Fig. 6. This is especially evident in the simulated column AOD over northern India and Beijing. For both ozone and AOD, comparison to the results from GCC at 4∘×5∘ shows that simulations with GCHP at coarse (C24) resolution are able to reproduce the same patterns, magnitude, and variability as those observed in GCC. Some differences are observed, such as the region over Afghanistan which shows an elevated ozone mixing ratio in GCC compared to GCHP. However, the results from GCHP at C24 show the same patterns, magnitude, and variability as those at 4∘×5∘.

Finally, we directly compare the simulated ozone data from GCHP and GCC using 2-D histograms. Figure 6 shows the ozone ratio in each 4∘×5∘ grid cell, as simulated in GCC (horizontal axis) and as simulated in GCHP (vertical axis) at two different horizontal resolutions. All data are binned at a resolution of 1 ppbv ozone. In each case, GCHP results are conservatively gridded to 4∘×5∘ to provide a direct comparison, and data in the “polar cap” regions (top and bottom two 4∘×5∘ latitude bands) are excluded. Fitting parameters are shown in white based on a reduced-major-axis (“geometric”) regression. Comparison of results at C24 to those at 4∘×5∘ shows good agreement, with a correlation coefficient r=0.88 and a slope of 0.95. At higher resolution, this agreement is slightly worsened (r=0.86 and slope of 0.92), as smaller-scale processes and chemical non-linearity are resolved which could not be represented at the coarser resolution.

Models of atmospheric chemistry have grown continuously in resolution and complexity over the past decades to take advantage of increasing computational resources. The GEOS-Chem High Performance (GCHP) model is a next step in this growth, enabling the widely used GEOS-Chem chemical transport model to exploit the computational speed and memory capacity of massively parallel architectures. In this manner, we can achieve routine simulation of global stratosphere–troposphere oxidant–aerosol chemistry at unprecedented resolution and detail.

Detailed documentation of GCHP including a user manual is available on the GCHP website (http://www.geos-chem.org, last access: 19 July 2018). GCHP incorporates the existing GEOS-Chem shared-memory code into an ESMF-based framework (MAPL), enabling GEOS-Chem to be run in a distributed memory framework across multiple nodes while retaining all the features of the high-fidelity global chemical simulation. In addition to a new model framework, GCHP replaces the conventional rectilinear latitude–longitude grid with the gnomonic cubed-sphere grid of the NASA GEOS meteorological data used as input to GEOS-Chem. This provides greater computational accuracy and efficiency for transport calculations while removing an additional restriction on scalability. GCHP performs with high computational scalability up to at least 540 cores, completing a 1-month simulation of oxidant–aerosol chemistry in the troposphere and stratosphere (206 active species, 135 tracers) at a global resolution of C180 (∼0.5∘×0.625∘) in 24 h.

GCHP also provides a mechanism for ongoing improvement of modeling capability. With the base GEOS-Chem model, GCHP, and the GMAO GEOS atmospheric data assimilation system now all using an identical copy of the grid-independent GEOS-Chem code, GCHP closes the loop between online and offline modelers, allowing seamless propagation of model and framework improvements between all three. Future development opportunities range from improved parallelism in input operations to the direct ingestion of archived mass fluxes to further improve transport calculation accuracy.