WRF-GC (v1.0): online coupling of WRF (v3.9.1.1) and GEOS-Chem (v12.2.1) for regional atmospheric chemistry modeling – Part 1: Description of the one-way model

We developed the WRF-GC model, an online coupling of the Weather Research and Forecasting (WRF) mesoscale meteorological model and the GEOS-Chem atmospheric chemistry model, for regional atmospheric chemistry and air quality modeling. WRF and GEOS-Chem are both open-source community models. WRF-GC offers regional modellers access to the latest GEOS-Chem chemical module, which is state of the science, well documented, traceable, benchmarked, actively developed by a large international user base, and centrally managed by a dedicated support team. At the same time, WRF-GC enables GEOS-Chem users to perform high-resolution forecasts and hindcasts for any region and time of interest. WRF-GC uses unmodified copies of WRF and GEOS-Chem from their respective sources; the coupling structure allows future versions of either one of the two parent models to be integrated into WRF-GC with relative ease. Within WRF-GC, the physical and chemical state variables are managed in distributed memory and translated between WRF and GEOS-Chem by the WRF-GC coupler at runtime. We used the WRF-GC model to simulate surface PM_2.5 concentrations over China during 22 to 27 January 2015 and compared the results to surface observations and the outcomes from a GEOS-Chem Classic nested-China simulation. Both models were able to reproduce the observed spatiotemporal variations of regional PM_2.5, but the WRF-GC model (r=0.68, bias =29 %) reproduced the observed daily PM_2.5 concentrations over eastern China better than the GEOS-Chem Classic model did (r=0.72, bias =55 %). This was because the WRF-GC simulation, nudged with surface and upper-level meteorological observations, was able to better represent the pollution meteorology during the study period. The WRF-GC model is parallelized across computational cores and scales well on massively parallel architectures. In our tests where the two models were similarly configured, the WRF-GC simulation was 3 times more efficient than the GEOS-Chem Classic nested-grid simulation due to the efficient transport algorithm and the Message Passing Interface (MPI)-based parallelization provided by the WRF software framework. WRF-GC v1.0 supports one-way coupling only, using WRF-simulated meteorological fields to drive GEOS-Chem with no chemical feedbacks. The development of two-way coupling capabilities, i.e., the ability to simulate radiative and microphysical feedbacks of chemistry to meteorology, is under way. The WRF-GC model is open source and freely available from http://wrf.geos-chem.org (last access: 10 July 2020).

1 Introduction

Regional models of atmospheric chemistry simulate the emission, transport, chemical evolution, and removal of atmospheric constituents over a given domain. These models are widely useful for forecasts of air quality, for impact assessment associated with polluting activities, and for theory validation by comparisons against observations. It is thus crucial that regional models be frequently updated to reflect the latest scientific understandings of atmospheric processes. At the same time, the increasing demand for fine-resolution simulations requires models to adapt to massively parallel computation architectures with high scalability. We present here the development of a new regional atmospheric chemistry model: WRF-GC, an online coupling of the Weather Research and Forecasting (WRF) mesoscale meteorological model and the GEOS-Chem atmospheric chemistry model, specifically designed to allow easy updates and be computationally efficient, for use in research and operation applications.

Regional atmospheric chemistry models fall into two categories: offline models and online models. Offline models (also called chemical transport models; CTMs) use archived meteorological fields, either those simulated by models alone or those assimilated with observations, to drive the transport and chemical evolution of atmospheric constituents (Baklanov et al., 2014). By eliminating the need to solve dynamical processes online, the developers of offline models can focus their efforts to solving more complex chemical processes. For example, one popular regional CTM is the GEOS-Chem model in its nested-grid configuration (Bey et al., 2001; Wang et al., 2004; Chen et al., 2009; Zhang et al., 2015), which is driven by high-resolution assimilated meteorological data from the GEOS model of the NASA Global Modeling and Assimilation Office (GMAO). GEOS-Chem has undergone three major chemical updates in the last year. Its latest standard chemical mechanism includes state-of-the-science O_x–NO_x–volatile organic compound (VOC)–halogen–aerosol reactions. In addition, GEOS-Chem offers a number of specialty simulations to address a variety of scientific questions, such as simulations of CO₂ (Nassar et al., 2010), CO (Fisher et al., 2017), methane (Maasakkers et al., 2019), mercury (Horowitz et al., 2017; Soerensen et al., 2010), persistent organic pollutants (Friedman et al., 2013), and dicarbonyls (Fu et al., 2008, 2009; Cao et al., 2018).

Despite their updated representation of chemical processes and relative ease of use, offline models have several key shortcomings. First, the applications of some offline models are limited by the time span and resolution of the available meteorological data. In the case of the GEOS-Chem nested-grid model, its application is currently limited to 0.5^∘ latitude $\times \mathrm{0.625}{}^{\circ }$ longitude or coarser resolution between 1979 and the present day when using the Modern-Era Retrospective analysis for Research and Applications, version 2 (MERRA-2) dataset, or to 0.25^∘ latitude $\times \mathrm{0.3125}{}^{\circ }$ longitude or coarser resolution between 2013 and the present day when using the GEOS Forward Processing (GEOS-FP) dataset. The temporal interpolation of sparsely archived meteorological data can also cause significant errors in the CTM simulations (Yu et al., 2018). Most importantly, offline models cannot simulate meteorology–chemistry interactions due to the lack of chemical feedback to meteorology.

In contrast, online regional atmospheric chemistry models perform integrated meteorological and chemical calculations, managed through operator splitting (Baklanov et al., 2014). In this way, online models can simulate regional atmospheric chemistry at any location and time of interest, without the need for temporal interpolation of the meteorological variables. Moreover, online models have the option to include “two-way coupling” processes, i.e., the response of meteorology to gases and aerosols via interactions with radiation and cloud processes. Many studies have demonstrated the importance of two-way interactions in accurate air quality simulations (e.g., Li et al. (2011); Ding et al. (2013); J. Wang et al. (2014)). One of the most widely used online regional models is WRF-Chem, with options for either one-way or two-way coupling (Grell et al., 2005; Fast et al., 2006). The latest version of WRF-Chem (v4.1) includes many options for O_x–NO_x–VOC–aerosol chemistry. WRF-Chem simulates the two-way interactions between chemistry and meteorology by taking into account the scattering and absorption of radiation by gases and aerosols, as well as the activation of aerosols as cloud condensation nuclei and ice nuclei (Fast et al., 2006; Gustafson et al., 2007; Chapman et al., 2009).

However, keeping the representation of atmospheric processes up to date is potentially more challenging for online models than it is for offline models. One of the reasons for this is that the interactions between the chemical and meteorological modules are hard wired in some online models, such that updating either module requires considerable effort. For the same reason, if users make improvements to the chemical or meteorological processes in the online model, those improvements may be relatively difficult to propagate to the broader community. This may lead to the model diverging into different branches, and users may be forced to work with stale, branched versions of the code.

In this work, we developed a new online regional atmospheric chemistry model, WRF-GC, by coupling the WRF meteorology model with the GEOS-Chem chemistry model. Both WRF and GEOS-Chem are open source and actively developed by the community. We constructed WRF-GC with the following guidelines, in order to best take advantage of new developments in the two parent models and to enhance usability:

1. The coupling structure of WRF-GC should be abstracted from the parent models, and both parent models remain unmodified from their respective sources. In this way, future updates of the parent models can be quickly incorporated into WRF-GC with ease, such that WRF-GC can stay cutting edge. It also enables WRF-GC users to more easily contribute their developments back to the parent models.

2. The WRF-GC coupled model should scale from conventional computation hardware to massively parallel computation architectures.

3. The WRF-GC coupled model should be easy to install and use, open source, version controlled, and well documented.

WRF-GC offers users of WRF-Chem or other regional models the option to use the latest GEOS-Chem chemical module, which is actively developed by a large international user base, well documented, traceable, benchmarked, and centrally managed. Through WRF-GC, regional modellers also gain access to the specialty simulations in GEOS-Chem, such as the simulations of mercury (Horowitz et al., 2017; Soerensen et al., 2010) and persistent organic pollutants (Friedman et al., 2013). WRF-GC drives GEOS-Chem with online meteorological fields simulated by WRF, which in turn can be driven by initial and boundary meteorological conditions from many different assimilated datasets or climate model outputs (Skamarock et al., 2008, 2019). As such, WRF-GC allows GEOS-Chem users to perform high-resolution simulations in both forecast and hindcast modes at any location and time of interest.

In this Part 1 paper, we describe the development of the WRF-GC model (v1.0, https://doi.org/10.5281/zenodo.3550330, Lin et al., 2019) for simulation over a single domain with one-way coupling capability. The nested domain and two-way coupling capabilities are under development and will be described in a forthcoming paper (Feng et al., 2020).

2 The parent models: WRF and GEOS-Chem

2.1 The WRF model

Meteorological processes and advection of atmospheric constituents in the WRF-GC coupled model are simulated by the WRF model (v3.9.1.1 or later versions), a mesoscale numerical weather model for research and operational applications (Skamarock et al., 2008, 2019). WRF offers its users many options for model configurations and physical schemes. WRF uses the Advanced Research WRF (ARW) dynamical solver, which solves fully compressible, Eulerian non-hydrostatic equations on either hybrid sigma–eta (default) or terrain-following vertical coordinates defined by the user. Horizontal grids in WRF are staggered Arakawa C grids, which can be configured by the user using four map projections: latitude–longitude, Lambert conformal, Mercator, and polar stereographic. WRF supports the use of multiple nested domains to simulate the interactions between large-scale dynamics and mesoscale meteorology. WRF supports grid, spectral, and observational nudging (Liu et al., 2005, 2006; Stauffer and Seaman, 1990, 1994). This allows the WRF model to produce meteorological outputs that mimic assimilated meteorological fields for use in air quality hindcasts. The WRF model offers many options for land surface physics, planetary boundary layer physics, radiative transfer, cloud microphysics, and cumulus parameterization, for use in meteorological studies, real-time numerical weather prediction, idealized simulations, and data assimilation on meso- to regional scales (Skamarock et al., 2008, 2019). Table 3 lists the configuration and physical options supported by WRF-GC v1.0. In particular, only the hybrid sigma–eta vertical coordinate is currently supported in WRF-GC.

Table 1 Meteorological variables needed for GEOS-Chem.

The WRF model incorporates a highly modular software framework that is portable across a range of computing platforms. WRF supports two-level domain decomposition for distributed-memory (Message Passing Interface; MPI) and shared-memory (OpenMP) parallel computation. Distributed parallelism is implemented through the Runtime System Library lite (RSL-lite) module, which supports irregular domain decomposition, automatic index translation, distributed input/output, and low-level interfacing with MPI libraries (Michalakes et al., 1999).

2.2 The GEOS-Chem model

Our development of WRF-GC was made possible by a recent structural overhaul of GEOS-Chem (Long et al., 2015; Eastham et al., 2018), which enabled the use of GEOS-Chem as a self-contained chemical module within the WRF-GC model. The original GEOS-Chem CTM (prior to v11.01) was structured specifically for several static sets of global or regional 3-D grids at prescribed horizontal and vertical resolutions (Bey et al., 2001). Parallelism for the original GEOS-Chem was implemented through OpenMP, such that the original GEOS-Chem can only operate on single-node hardware with large shared memory. Long et al. (2015) restructured the core processes in GEOS-Chem, including emission, chemistry, convective mixing, planetary boundary layer mixing, and deposition processes, to work in modular units of atmospheric vertical columns. Information about the horizontal grids, formerly fixed at compile time, is now passed to the GEOS-Chem chemical module at runtime. This development enabled the use of the GEOS-Chem chemical module with any horizontal grid structure and horizontal resolution.

The new, modularized structure of the GEOS-Chem has been implemented in two types of configurations. The first type of configuration uses GEOS-Chem as the core of offline CTMs. For example, in the GEOS-Chem “Classic” implementation (GCC), the GEOS-Chem chemical module is driven by the GEOS assimilated meteorological data and is parallelized using OpenMP. This implementation treats the prescribed global or regional model domains as contiguous sets of atmospheric columns placed at regular latitude/longitude intervals, with vertical layers pre-defined to match those of the GEOS model. In essence, the GCC implementation mimics the “original” GEOS-Chem model before the structural overhaul by Long et al. (2015). Other grid systems can also be used with the GEOS-Chem chemical module. For example, the GEOS-Chem High Performance (GCHP) implementation (Eastham et al., 2018) calls the GEOS-Chem chemical module on the native cubed-sphere coordinates of the NASA GEOS model via a column interface in GEOS-Chem (GIGC_Chunk_Run). This column interface was built on the Earth System Modeling Framework (ESMF) (Eastham et al., 2018) and permits runtime specification of the horizontal grid parameters. The GCHP implementation uses MPI to parallelize GEOS-Chem across nodes through the Model Analysis and Prediction Layer framework (MAPL) (Suarez et al., 2007), which is a wrapper on top of ESMF specifically designed for the NASA GEOS model system.

Alternatively, GEOS-Chem can be used as a module coupled to weather models or Earth system models to perform online chemical calculations. Using this capability, Hu et al. (2018) developed an online implementation of GEOS-Chem by coupling it to the NASA GEOS-5 model to simulate global atmospheric chemistry. Lu et al. (2019) coupled GEOS-Chem to the Beijing Climate Center Atmospheric General Circulation Model (BCC-AGCM). However, both the NASA GEOS-5 model and the BCC-AGCM are proprietary.

WRF-GC is the first implementation that couples the GEOS-Chem chemical module to an open-access high-resolution meteorological model. We developed a modular coupler between WRF and GEOS-Chem that draws from the technology of GCHP but does not rely on ESMF (described in Sect. 3.2). We also made changes to GEOS-Chem to accept arbitrary vertical discretization from WRF at runtime and to improve physical compatibility with WRF (described in Sect. 3.2.1). These changes have been incorporated into the mainline GEOS-Chem code. Our coupler and code modifications can be adapted in the future to couple GEOS-Chem to other non-ESMF Earth system models.

Chemical calculations in WRF-GC v1.0 use GEOS-Chem version 12.2.1 (https://doi.org/10.5281/zenodo.2580198, The International GEOS-Chem Community, 2019). The standard chemical mechanism in GEOS-Chem v12.2.1, used by default in WRF-GC, includes detailed O_x–NO_x–VOC–ozone–halogen–aerosol in the troposphere, as well as the unified tropospheric–stratospheric chemistry extension (UCX) (Eastham et al., 2014) for stratospheric chemistry and stratosphere–troposphere exchange. The standard gas-phase mechanism includes 208 chemical species and 981 reactions. Reactions and rates follow the latest recommendations from the Jet Propulsion Laboratory and the International Union of Pure and Applied Chemistry. In addition, GEOS-Chem uses the FlexChem pre-processor (a wrapper for the Kinetic PreProcessor, KPP; Damian et al., 2002; Sandu and Sander, 2006) to configure chemical kinetics (Long et al., 2015). This allows users to add or modify gaseous species and reactions to develop custom mechanisms and diagnostic quantities in GEOS-Chem. GEOS-Chem also supports the optional “Tropchem” (troposphere-only chemistry) mechanism, where UCX is disabled and replaced by a parameterized linear chemistry in the stratosphere (McLinden et al., 2000). GEOS-Chem uses the Rosenbrock Rodas3 solver (Sandu et al., 1997) for gas-phase chemistry by default, but users may choose other solvers through KPP.

Aerosol species in GEOS-Chem include secondary inorganic aerosols (sulfate, nitrate, ammonium), elemental carbon aerosol (EC), primary organic carbon (POC), secondary organic aerosol (SOA), dust, and sea salt. By default, secondary inorganic aerosols, EC, POC, and SOA are simulated as speciated bulk masses. Dust aerosols are represented in 4 size bins (0.1–1.0, 1.0–1.8, 1.8–3.0, and 3.0–6.0 µm) (Fairlie et al., 2007), while sea salt aerosols are represented in two size bins (0.1–0.5 and 0.5–4.0 µm) (Jaeglé et al., 2011). The thermodynamics of secondary inorganic aerosol are coupled to gas-phase chemistry and computed by the ISORROPIA II module (Park et al., 2004; Fountoukis and Nenes, 2007; Pye et al., 2009). EC and POC are represented in GEOS-Chem as partially hydrophobic and partially hydrophilic, with a conversion timescale from hydrophobic to hydrophilic of 1.2 d (Q. Wang et al., 2014). The organic matter to organic carbon (OM ∕ OC) mass ratio is assumed to be 2.1 for POC by default, with an option to use seasonally and spatially varying OM ∕ OC ratios (Philip et al., 2014). GEOS-Chem has two options to describe the production of SOA, and both options are supported in WRF-GC. By default, SOA is produced irreversibly using simple yields from anthropogenic and biogenic volatile organic precursors (Kim et al., 2015). Alternatively, GEOS-Chem can simulate SOA production via the aging of semi-volatile and intermediate volatility organic precursors using a volatility basis set (VBS) scheme (Robinson et al., 2007; Pye et al., 2010), as well as via the aqueous reactions of the oxidation products from isoprene (Marais et al., 2016). The GEOS-Chem model also has the option of simulating detailed, size-dependent aerosol microphysics using the TwO-Moment Aerosol Sectional microphysics (TOMAS) module (Kodros and Pierce, 2017) or the Advanced Particle Microphysics (APM) module (Yu and Luo, 2009), but these two modules are not yet supported in WRF-GC.

Emissions of chemical species in WRF-GC are calculated using the Harvard-NASA Emissions Component (HEMCO) in GEOS-Chem (Keller et al., 2014). HEMCO allows users to select emission inventories from the HEMCO data directory or add their own inventories, and interpolate the emission fluxes to the model domain and resolution at runtime. The HEMCO data directory currently includes more than 20 global and regional emission inventories, mostly at their respective native resolutions (http://wiki.seas.harvard.edu/geos-chem/index.php/HEMCO_data_directories, last access: 24 April 2020). By default, the Community Emissions Data System (CEDS) inventory (${\mathrm{0.5}}^{\circ }\times {\mathrm{0.5}}^{\circ }$ resolution, monthly) (Hoesly et al., 2018) is used for most of the world; over Asia and the US, the CEDS is superseded by the MIX inventory (${\mathrm{0.25}}^{\circ }\times {\mathrm{0.25}}^{\circ }$ native resolution, monthly; Li et al., 2017b) and the 2011 National Emission Inventory (NEI 2011) (0.1 km×0.1 km native resolution, hourly; U.S. Environmental Protection Agency, 2014), respectively. HEMCO also has extensions to compute emissions with meteorological dependencies, such as the emissions of biogenic species (Guenther et al., 2012), soil NO_x (Hudman et al., 2012), lightning NO_x, sea salt (Gong, 2003), and dust (Zender et al., 2003). With the exception of lightning NO_x, these meteorology-dependent emissions are supported in WRF-GC v1.0. Further details about the use of HEMCO in WRF-GC are given in Sect. 3.3.1.

Other physical calculations in GEOS-Chem are coupled to WRF meteorological fields in WRF-GC; we describe the coupling in detail in Sect. 3.3. Convective transport of chemical species is calculated using a single-plume parameterization (Allen et al., 1996; Wu et al., 2007), which is in turn driven by the cumulus parameterization in WRF. Boundary layer mixing is calculated using a non-local scheme, driven by the WRF-simulated atmospheric instability and boundary layer height (Lin and McElroy, 2010). Dry deposition is based on a resistance-in-series scheme (Wesely, 1989; Wang et al., 1998). Aerosol deposition is as described in Zhang et al. (2001), with updates to account for size dependency for dust (Fairlie et al., 2007) and sea salt (Alexander et al., 2005; Jaeglé et al., 2011). Wet scavenging of gases and water-soluble aerosols is as described in Liu et al. (2001) and Amos et al. (2012) and driven by WRF-simulated precipitation rates.

3 Description of the WRF-GC coupled model

3.1 Overview of the WRF-GC model architecture

Figure 1 gives an architectural overview of the WRF-GC coupled model. Our development of WRF-GC uses many of the existing infrastructure in the WRF-Chem model that couples WRF to its chemistry module (Grell et al., 2005). The subroutine calls between WRF and the chemistry components are exactly the same in WRF-GC and in WRF-Chem. Within WRF-GC, the WRF model and the GEOS-Chem model remain entirely intact. The WRF-GC coupler consists of interfaces with the two parent models, as well as a state conversion module and a state management module. The coupler is separate from both parent models and is written in a manner similar to an application programming interface, enabling independent updates of the parent models within WRF-GC.

Figure 1 Architectural overview of the WRF-GC model (v1.0). The WRF-GC coupler (all parts shown in red) includes interfaces to the two parent models, as well as the state conversion and state management modules. The parent models (shown in grey) are standard codes downloaded from their sources, without any modifications.

The WRF-GC model is initialized and driven by WRF, which sets up the simulation domain, establishes the global clock, handles the input and output of data, sets the initial and boundary conditions (IC/BC) for meteorological and chemical variables, and manages cross-processor communication for parallelization. Users configure the WRF-GC model structure in the WRF configuration file (namelist.input), including the domain, projection, horizontal resolution, vertical coordinates, simulation time, dynamic time step, external chemical time step, and other physical and dynamical options. Users also “turn on” GEOS-Chem in WRF-GC by specifying chem_opt = 233 in namelist.input, similar to the way that users specify the chemical mechanism in WRF-Chem. GEOS-Chem is initialized by the WRF model using the WRF-to-chemistry interface described in Sect. 3.2.3. Chemical options within GEOS-Chem, including the choice of standard or custom chemical mechanisms, emission inventories in HEMCO, and diagnostic quantities to be output, are defined by users in the GEOS-Chem configuration files (input.geos, HEMCO_Config.rc, and HISTORY.rc). IC/BC for meteorological and chemical variables are prepared by the user in NetCDF format and read by WRF. Meteorological IC/BC can be prepared using the WRF pre-processor system (WPS) from datasets available from NCAR's Research Data Archive (https://rda.ucar.edu, last access: 24 April 2020). IC/BC of chemical species concentrations are taken from GEOS-Chem Classic global model outputs and interpolated to the WRF-GC model grids using a modified version of the WRF-Chem pre-processor tool mozbc (available along with the WRF-GC code).

Dynamical and physical calculations are performed in WRF-GC exactly as they are in the WRF model. WRF also calculates the grid-scale advection of chemical species. At the beginning of each chemical time step, WRF calls the WRF-GC chemistry component through the WRF-to-chemistry interface, chem_driver, which is a chemical driver similar to that in WRF-Chem. Spatial parameters and the internal state of WRF are translated at runtime to GEOS-Chem by the state conversion and management modules. The GEOS-Chem chemical module then performs convective transport, dry deposition, wet scavenging, emission calculations, boundary layer mixing, and chemistry calculations. This operator splitting between WRF and GEOS-Chem is identical to that between WRF and the chemistry module in WRF-Chem. Then, the GEOS-Chem internal state is translated back to WRF, and the WRF time stepping continues. At the end of the WRF-GC simulation, WRF outputs all meteorological and chemical diagnostic quantities in WRF's standard format.

WRF-GC v1.0 supports all the existing input and output functionality of the WRF model, including serial/parallel reading and writing of NetCDF, HDF5, and GRIB2 datasets. WRF and WRF-Chem users can use existing data pre- and post-processing tools to prepare input data and analyze results from WRF-GC.

3.2 Details about the WRF-GC coupler technology

3.2.1 Further modularization of GEOS-Chem for WRF-GC coupling

Long et al. (2015) restructured the GEOS-Chem model into modular units of atmospheric columns. However, there were limitations in that column structure and its interface which prohibit the coupling with WRF. First, the GEOS-Chem module developed by Long et al. (2015) was hard-coded to operate on prescribed configurations of either 72 or 47 vertical levels. The former configuration was designed to match the native vertical levels of the GEOS model. The latter configuration was designed to match the lumped vertical levels often used by the GEOS-Chem Classic model. Second, the column interface to the GEOS-Chem module as implemented in GCHP depends on the ESMF and MAPL frameworks, which WRF does not support.

We modified the GEOS-Chem module and interface to facilitate more flexible coupling with WRF and other dynamical models. We allowed GEOS-Chem to accept the A_p and B_p parameters for the hybrid sigma–eta vertical grids and the local tropopause level from WRF at runtime. Stratospheric chemistry is only calculated in GEOS-Chem above the tropopause level passed from WRF. Also, 3-D emissions (such as the injection of biomass burning plumes into the free troposphere) are interpolated in HEMCO to the WRF-GC vertical levels.

In addition, we modified the existing GCHP interface GIGC_Chunk_Run to remove its dependencies on ESMF and MAPL when running in WRF-GC. We added a set of compatible error-handling and state management components to GEOS-Chem that interacts with the WRF through the chemistry driver. These new components replace the functionalities originally provided by ESMF. This removes all dependency of the WRF-GC coupler and the GEOS-Chem column interface on external frameworks.

All of our changes adhere to the GEOS-Chem coding and documentation standards and have been fully merged into the GEOS-Chem standard source code as of version 12.0.0 (https://doi.org/10.5281/zenodo.1343547, The International GEOS-Chem Community, 2019) and are controlled with the pre-processor switch MODEL_WRF at compile time. In the future, these changes will be maintained as part of the standard GEOS-Chem model.

3.2.2 Installation and compilation process

From the user's standpoint, the installation and compilation of WRF-GC is very similar to the procedures for WRF-Chem. WRF-GC is installed by downloading the parent models, WRF and GEOS-Chem, and the WRF-GC coupler, directly from their respective software repositories. The WRF model is installed in a top-level directory, while the WRF-GC coupler and GEOS-Chem are installed under the chem/ subdirectory, where the chemistry routines for WRF-Chem originally reside. An unmodified copy of the GEOS-Chem code is installed in the chem/gc/ subdirectory, and a set of sample GEOS-Chem configuration files is in chem/config/. The WRF meteorology model remains unmodified in WRF-GC, but at present the chemical routines of WRF-Chem cannot work alongside GEOS-Chem under a single WRF-GC top directory.

The standard WRF model includes built-in compile routines for coupling with the WRF-Chem chemistry module. WRF-GC uses these existing compile routines by substituting the parts pertinent to WRF-Chem with a generic chemistry interface. This substitution process is self-contained in the WRF-GC coupler and requires no manual changes to the WRF code. As such, the installation and compilation of WRF-GC require no extra maintenance effort from the WRF developers, and WRF-GC operates as a drop-in chemical module to WRF.

When the user sets the environment variable WRF_CHEM to 1 in the WRF compile script, WRF reads a registry file (registry.chem) containing the GEOS-Chem chemical species information (duplicated from input.geos) and builds these species into the WRF model framework. The WRF compile script then calls the Makefile in the chem/ subdirectory to compile an unmodified copy of GEOS-Chem (located in chem/gc/) with the pre-processor switch MODEL_WRF. This compiles GEOS-Chem into two libraries, which can be called by WRF. The first GEOS-Chem library (libGeosCore.a) contains all GEOS-Chem core routines. The second GEOS-Chem library (libGIGC.a) contains the GEOS-Chem column interface (GIGC_Chunk_Mod). The subsequent compilation process links these GEOS-Chem libraries and the WRF-to-chemistry interface to the rest of the WRF code, creating a single WRF-GC executable (wrf.exe).

3.2.3 Runtime processes

In WRF-Chem, WRF calls its interface to chemistry, chem_driver, which then calls each individual chemical processes. We abstracted this chem_driver interface by removing direct calls to chemical processes. Instead, our chem_driver calls the WRF-GC state conversion module (WRFGC_Convert_State_Mod) and the GEOS-Chem column interface (GIGC_Chunk_Run) to perform chemical calculations. We also modified chemics_init to initializes GEOS-Chem through the column interface GIGC_Chunk_Init.

The WRF-GC state conversion module includes two subroutines. The WRFGC_Get_WRF subroutine receives meteorological data and spatial information from WRF and translates them into GEOS-Chem formats and units. Table 1 summarizes the WRF meteorological variables required to drive GEOS-Chem. Many meteorological variables in WRF only require a conversion of units before they are passed to GEOS-Chem. Some meteorological variables require physics-based diagnosis in the WRFGC_Get_WRF subroutine before being passed to GEOS-Chem. Horizontal grid coordinates and resolutions are passed to GEOS-Chem in the form of latitudes and longitudes at the center and edges of each grid. Vertical coordinates are passed from WRF to GEOS-Chem at runtime as described in Sect. 3.2.1. A second subroutine, WRFGC_Set_WRF, receives chemical species concentrations from GEOS-Chem, converts the units, and saves them in the chemical variable array in WRF.

The WRF-GC state management module (GC_Stateful_Mod) manages the GEOS-Chem internal state in distributed memory, such that GEOS-Chem can run in the MPI-parallel architecture. When running WRF-GC in the distributed-memory configuration, WRF decomposes the horizontal computational domain evenly across the available computational cores at the beginning of runtime. Each computational core has access only to its allocated subset of the full domain as a set of atmospheric columns, plus a halo of columns around that subset domain. The halo columns are used for inter-core communication of grid-aware processes, such as horizontal transport (Skamarock et al., 2008). The internal states of GEOS-Chem for each core are managed by the state management module; they are distributed at initialization and independent from each other. The WRF-GC state management module is also critical to the development of nested-grid simulations in the future.

3.3 Treatment of key processes in the WRF-GC coupled model

Below, we describe the operator splitting between WRF and GEOS-Chem within WRF-GC, as well as the treatments of some of the key processes. The general Eulerian form of the continuity equation for m chemical species with number density vector $\mathit{n}=(n_{\mathrm{1}},\mathrm{\dots },n_m{)}^T$ is

$$\begin{array}{}\text{(1)}& {\displaystyle \frac{\partial n_i}{\partial t}}=-\mathrm{\nabla }\cdot \left(n_i\mathit{U}\right)+P_i\left(\mathit{n}\right)+L_i\left(\mathit{n}\right)i\in [\mathrm{1},m].\end{array}$$

U is the wind vector, which is provided by the WRF model in WRF-GC. The first term on the right-hand side of Eq. (1) indicates the transport of species i, which includes grid-scale advection, as well as subgrid turbulent mixing and convective transport. P_i(n) and L_i(n) are the local production and loss rates of species i, respectively (Long et al., 2015).

In WRF-GC, WRF simulates the meteorological variables using the dynamic equations and meteorological IC/BC. These meteorological variables are then passed to the GEOS-Chem chemical module (Table 1) to solve the local production and loss terms of the continuity equation. Large-scale (grid-scale) advection of chemical species is grid-aware and is calculated by the WRF dynamical core. Local (subgrid) vertical transport processes, including turbulent mixing within the boundary layer and convective transport from the surface to the convective cloud top, are calculated in GEOS-Chem using WRF-simulated meteorology. Dry deposition and wet scavenging of chemical species are also calculated in GEOS-Chem and driven by WRF. This operator-splitting arrangement is identical to that in the WRF-Chem model.

The dynamic and chemical time steps are specified by the user in the WRF configuration file namelist.input. The dynamic time step is constrained by the Courant–Friedrichs–Lewy stability criterion and should be short for high-resolution simulations. WRF-Chem recommends that the chemical time step be set the same as the dynamic time step as best practice (Peckham et al., 2017). Because GEOS-Chem uses a Rosenbrock solver, which adapts its internal chemical time step to the stiffness of the chemical mechanism, a larger chemical time step may be used. However, it is recommended that the results be compared to a control simulation with the chemical time step set to the dynamic time step (Peckham et al., 2017).

3.3.1 Emission of chemical species

Chemical emissions in WRF-GC are calculated online by the HEMCO module in GEOS-Chem (Keller et al., 2014) and configured in HEMCO_Config.rc. HEMCO and its data directory are updated as part of the GEOS-Chem model and remain unmodified in WRF-GC. Users can choose to use one or combine several of the emission inventories already in the HEMCO data directory (Sect. 2.2). Some of the inventories currently available in the HEMCO data directory may not be of sufficiently fine resolution to support the high-resolution WRF-GC simulations. In that case, users can prepare their own emission input files in NetCDF format at arbitrary spatiotemporal resolutions, and HEMCO will interpolate them to the WRF-GC model domain and resolution at runtime. HEMCO also allow users to specify scale factors and diurnal/weekly/monthly variation profiles in HEMCO_Config.rc to be applied to the emission fluxes at runtime. WRF-GC calls HEMCO to compute meteorology-dependent emissions online using WRF-simulated meteorology. These currently include the emissions of biogenic species (Guenther et al., 2012), soil NO_x (Hudman et al., 2012), sea salt (Gong, 2003), and dust (Zender et al., 2003). Lightning NO_x emissions are not yet supported in WRF-GC v1.0 but will be added in a future version.

3.3.2 Subgrid vertical transport of chemical species

Subgrid vertical transport of chemical species in WRF-GC, including convective transport and boundary layer mixing, are calculated by GEOS-Chem using WRF meteorology. GEOS-Chem uses the convective mass flux variable to drive convective transport (Allen et al., 1996; Wu et al., 2007). This variable is calculated in WRF using the user's choice of cumulus parameterization, but its value is not stored in the WRF meteorological variable array. The WRF-GC state conversion module re-diagnoses convective mass fluxes using the user's choice of parameterization in WRF and then pass the values to GEOS-Chem. The WRF-GC state conversion module currently supports convective mass flux calculations using the new Tiedtke scheme (Tiedtke, 1989; Zhang et al., 2011; Zhang and Wang, 2017) and the Zhang–McFarlane scheme (Zhang and McFarlane, 1995) (Table 1), because these two cumulus parameterization schemes are more physically compatible with the convective transport algorithm currently in GEOS-Chem. In addition, the users should consider the horizontal resolution of the model when choosing which cumulus parameterization to use. The new Tiedtke scheme and the Zhang–McFarlane schemes are generally recommended for use in simulations at horizontal resolutions larger than 10 km (Skamarock et al., 2008; Arakawa and Jung, 2011). At horizontal resolutions between 2 and 10 km, the so-called “convective grey zone” (Jeworrek et al., 2019), the use of the Grell–Freitas scheme is recommended for the WRF model (Grell and Freitas, 2014), as it allows subsidence to spread to neighboring columns; this option will be implemented in a future WRF-GC version. At horizontal resolutions finer than 2 km, it is assumed that convections are resolved and cumulus parameterizations should not be used (Grell and Freitas, 2014; Jeworrek et al., 2019). The scale dependence of cumulus parameterizations and their impacts on convective mixing of chemical species are an active area of research, which we will explore in the future using WRF-GC.

Boundary layer mixing is calculated in GEOS-Chem using a non-local scheme (Holtslag and Boville, 1993; Lin and McElroy, 2010). The boundary layer height, thermodynamic variables, and the vertical level and pressure information are calculated by WRF and passed to GEOS-Chem through the state conversion module. Again, this methodology is the same as that in the WRF-Chem model.

3.3.3 Dry deposition and wet scavenging of chemical species

Dry deposition is calculated in GEOS-Chem using a resistance-in-series scheme (Wesely, 1989; Wang et al., 1998). The land cover data for the simulated domain are read by and used in WRF, but for now WRF-GC only supports the use of the US Geological Survey (USGS) classification. The land cover information is passed to GEOS-Chem, where it is mapped to the land cover classifications of Olson et al. (2001) to assign values of surface roughness and canopy resistance (Wang et al., 1998). The dry deposition velocities are calculated locally using WRF-simulated surface air momentum, sensible heat fluxes, temperature, and solar radiation.

To calculate the wet scavenging of chemical species, the WRF-GC coupler diagnoses the WRF-simulated precipitation variables using the microphysical scheme and cumulus parameterization scheme selected by the user (Table 1). The precipitation variables passed to GEOS-Chem include large-scale/convective precipitation production rates, large-scale/convective precipitation evaporation rates, and the downward fluxes of large-scale and convective ice/liquid precipitation. The microphysical schemes currently supported in WRF-GC include the Morrison two-moment scheme (Morrison et al., 2009), the CAM5.1 scheme (Neale et al., 2012), the WSM6 scheme (Hong and Lim, 2006), and the Thompson scheme (Thompson et al., 2008).

4 Application: surface PM_2.5 over China during 22 to 27 January 2015

We simulated surface PM_2.5 concentrations over China during a severe haze event in January 2015 using both the WRF-GC model v1.0 (WRF v3.9.1.1 and GEOS-Chem v12.2.1) and the GEOS-Chem Classic model (v12.2.1) in its nested-grid configuration for China. We compared the simulated results from the two models against each other, as well as against surface measurements. Both the WRF-GC and the GEOS-Chem Classic nested-China simulations were conducted from 18 to 27 January 2015; the first 4 d initialized the models. Results from 22 to 27 January 2015 were analyzed. Our goal was to compare the performance of the two models in simulating Chinese surface PM_2.5 under their normal mode of operation. To this end, the two simulations were configured as similarly as possible, but there are important innate differences between the two models, as described below.

4.1 Setup of the WRF-GC model and the GEOS-Chem model

Figure 2a shows the domain of the GEOS-Chem Classic nested-China simulation, which was driven by the GEOS-FP assimilated meteorological dataset at its native horizontal resolution of 0.25^∘ latitude $\times \mathrm{0.3125}{}^{\circ }$ longitude. The vertical resolution of the GEOS-FP dataset was reduced from its native 72 levels to 47 levels by lumping levels in the stratosphere. This treatment is standard in GEOS-Chem Classic simulations. The 47 vertical layers extended from the surface to 0.01 hPa, with seven levels in the bottom 1 km. The top edge of the bottom layer was at approximately 120 m over eastern China. Meteorological variables were updated every 3 h (every hour for surface variables). IC/BC of chemical species concentrations were taken from the outputs of a global GEOS-Chem Classic simulation and updated at the boundaries of the nested-China domain every 3 h. The dynamic time step and the external chemistry time step were 5 and 10 min, respectively.

Figure 2 Comparison of the simulated (filled contours) 6 d average PM_2.5 concentrations during 22 to 27 January 2015 from (a) the GEOS-Chem Classic nested-China simulation and (b) the WRF-GC nudged simulation. Also shown are the observed 6 d average PM_2.5 concentrations during this period at 578 surface sites managed by the Ministry of Ecology and Environment of China.

Figure 2b shows the WRF-GC simulation domain, with a horizontal resolution of 27 km×27 km. We chose this domain and horizontal resolution to be comparable to those of the GEOS-Chem Classic nested-grid simulation. There were 50 vertical levels in our WRF-GC simulation, which extended from the surface up to 10 hPa with 7 levels below 1 km. The top edge of the bottom layer was at approximately 60 m over eastern China. The WRF model does not have the option of using the GEOS-FP dataset for meteorological IC/BC. Instead, we used the NCEP FNL dataset (https://doi.org/10.5065/D6M043C6, National Centers for Environmental Prediction et al., 2000) at $\mathrm{1}{}^{\circ }\times \mathrm{1}{}^{\circ }$ resolution as IC/BC for WRF-GC; the FNL dataset was interpolated to WRF vertical levels and updated every 6 h. In addition, we nudged the WRF-simulated meteorological fields with surface (every 3 h) and upper air (every 6 h) observations of temperature, specific humidity, and winds from the NCEP ADP Global Surface/Upper Air Observational Weather Database (https://doi.org/10.5065/39C5-Z211, Satellite Services Division et al., 2004). This mimicked the effect of meteorological data assimilation and allowed the WRF-simulated meteorology to stay close to the observed states of the atmosphere. IC/BC of chemical species concentrations were identical to those used in the GEOS-Chem Classic nested-China simulation but interpolated to WRF vertical levels and updated every 6 h. The dynamic time step and the external chemistry time step were 2 and 10 min, respectively. Other physical options used in our WRF-GC simulation are summarized in Table 2.

(Morrison et al., 2009)(Iacono et al., 2008)(Iacono et al., 2008)(Jimenez et al., 2012)(Chen and Dudhia, 2001a, b)(Nakanishi and Niino, 2006)(Tiedtke, 1989; Zhang et al., 2011; Zhang and Wang, 2017)

Table 2 WRF-GC physics configuration.

Our WRF-GC and GEOS-Chem Classic simulations used the exact same GEOS-Chem chemical mechanism for gases and aerosols, including a total of 241 chemical species. Emissions in the two simulations were both calculated by HEMCO and were identical for anthropogenic and biomass burning sources. Monthly mean anthropogenic emissions from China were from the Multi-resolution Emission Inventory for China (MEIC; Li et al., 2014) at $\mathrm{0.25}{}^{\circ }\times \mathrm{0.25}{}^{\circ }$ horizontal resolution. The MEIC inventory was updated for the year 2015 and included emissions from power generation, industry, transportation, and residential activities. Sector-specific weekly and diurnal variations from the MEIC inventory were applied (Li et al., 2017a). Agricultural emissions of ammonia were from Huang et al. (2012). Anthropogenic emissions from the rest of the Asia were from Li et al. (2017b), developed for the year 2010. Monthly mean biomass burning emissions were taken from Global Fire Emissions Database version 4 (GFED4) (Randerson et al., 2018). Emissions of biogenic species (Guenther et al., 2012), soil NO_x (Hudman et al., 2012), sea salt (Gong, 2003), and dust (Zender et al., 2003) in the two simulations were coupled to model meteorology in HEMCO and thus different between the two models. PM_2.5 mass concentrations were diagnosed for both simulations as the sum of masses of sulfate, nitrate, ammonium, EC, primary organic aerosol (assumed to be 2.1 times the primary organic carbon mass), SOA, fine dust (100 % of dust between 0 and 0.7 µm and 38 % of dust between 0.7 and 1.4 µm), and accumulation-mode sea salt, taking into consideration the hygroscopic growth for each species at 35 % relative humidity.

Table 3 List of WRF configuration and physical options supported in WRF-GC v1.0.

4.2 Validation of the WRF-GC simulation against surface measurements and comparison with the GEOS-Chem Classic simulation

Figure 2 compares the 6 d average surface PM_2.5 concentrations during 22 to 27 January 2015 as in the WRF-GC and the GEOS-Chem Classic nested-China simulations, respectively. Also shown are the PM_2.5 concentrations measured at 578 surface sites, managed by the Ministry of Ecology and Environment of China (http://www.cnemc.cn, last access: 24 April 2020). We removed invalid hourly surface PM_2.5 observations following the protocol in Jiang et al. (2020). The 578 sites were selected by (1) removing surface sites with less than 80 % valid hourly measurements during our simulation period, and (2) sampling the site closest to the model grid center, if that model grid contained multiple surface sites. Both models reproduced the general spatial distributions of the observed PM_2.5 concentrations, including the high concentrations over eastern China, as well as the hotspots over the North China Plan, central China, and the Sichuan Basin. However, both models overestimated the PM_2.5 concentrations over eastern China. The mean 6 d PM_2.5 concentrations averaged for the 578 sites as simulated by WRF-GC and by GEOS-Chem Classic nested-China were 117±68 and 120±76 µg m⁻³, respectively. In comparison, the observed mean 6 d PM_2.5 concentration averaged for the 578 sites was 98±43 µg m⁻³.

Figure 3 Scatter plots of observed and simulated daily mean PM_2.5 during 22 to 27 January 2015 at 507 surface sites over eastern China for (a) the GEOS-Chem Classic nested-China simulation and (b) the WRF-GC nudged simulation. The solid lines indicate the reduced major-axis regression lines, with slopes, intercepts, and correlation coefficients (r) shown in the inset. The dotted lines indicate the 1:1 lines.

Figure 3 shows the scatter plots of the simulated and observed daily average PM_2.5 concentrations over eastern China (eastward of 103^∘ E, 507 sites) during 22 to 27 January 2015. We focused on eastern China, because the spatiotemporal variability of PM_2.5 concentrations was higher over this region. Again, both models overestimated the daily PM_2.5 concentrations over eastern China. The daily PM_2.5 concentrations simulated by WRF-GC were 29 % higher than the observations (quantified by the reduced major-axis regression slope between the simulated and observed daily PM_2.5 concentration), with a correlation coefficient of r=0.68. The daily PM_2.5 concentrations in the GEOS-Chem Classic nested-China simulation were 55 % higher than the observations, with a correlation coefficient of r=0.72.

Figure 4 Taylor diagrams of hourly PM_2.5 concentrations during 22 to 27 January 2015 from the GEOS-Chem Classic nested-China simulation (black symbols) and the WRF-GC simulation (red symbols) for (a) 13 cities in the Beijing–Tianjin–Hebei area, (b) 22 cities in the Yangtze River Delta area, and (c) 13 other major Chinese cities. Dashed green, black, and blue lines indicate contours of the normalized centered root mean square differences (RMSDs), the ratios of simulated versus observed standard deviation, and the Pearson correlation coefficients, respectively.

Figure 4 shows the Taylor diagrams of the hourly PM_2.5 concentrations simulated by the two models at 48 major eastern Chinese cities, including 13 cities in the Beijing–Tianjin–Hebei (BTH) area, 22 cities in the Yangtze River Delta (YRD) area, and 13 other major cities. The Taylor diagram (Taylor, 2001) evaluates the simulated time series of PM_2.5 against the observations, using the Pearson correlation coefficients, the ratio between the simulated and observed standard deviations ($\frac{{\mathit{\sigma }}_{\text{sim}}}{{\mathit{\sigma }}_{\text{obs}}}$), and the normalized root mean square differences (RMSDs) as metrics. Proximity to the point “1” on the x axis in Fig. 4 indicates that the simulation accurately reproduced both the mean concentration and the temporal variability of the observations. For most cities in the BTH area and for most of the other 13 major cities, the hourly PM_2.5 concentrations simulated by WRF-GC showed smaller RMSDs and higher correlation coefficients against observations, compared to those in the GEOS-Chem Classic nested-China simulation. In the YRD area, the performance of the two models was similar.

Our analyses above show that the hourly and daily surface PM_2.5 concentrations simulated by the WRF-GC model were in better agreement with observations than those simulated by the GEOS-Chem Classic nested-China model over eastern China during 22 to 27 January 2015. We found that this was partially because the WRF-GC model, nudged with surface and upper-air meteorological observations, better represented the pollution meteorology, compared to the GEOS-FP dataset that was used to drive the GEOS-Chem Classic nested-China simulation. Figure S1 shows the average surface air temperature, relative humidity, and 10 m wind speed as simulated by the WRF-GC model and as provided by the GEOS-FP dataset against the observations during 22 and 27 January 2015 at 367 sites over China. The surface air temperature simulated by WRF-GC and those in the GEOS-FP dataset were both in good agreement with the observations over China. However, the relative humidity and wind speeds simulated by WRF-GC were more consistent with the observations, compared to those in the GEOS-FP dataset. Figure S2 assesses the hourly surface air temperature, relative humidity, and near-surface winds simulated by the WRF-GC model and those in the GEOS-FP assimilated meteorological dataset, against hourly surface measurements over China during 22–27 January 2015. For the 34 sites with publicly available hourly measurements, the meteorological fields simulated by the WRF-GC were generally more consistent with the measurements.

Figure 5 shows the mean planetary boundary layer height (PBLH) at 08:00 LT (00:00 UTC) and 20:00 LT (12:00 UTC) during 22 to 27 January 2015 in the GEOS-Chem Classic nested-China and the WRF-GC simulations, respectively, and compares them with the rawinsonde observations during this period (Guo et al., 2016). The PBLH in the GEOS-Chem Classic model was taken from the GEOS-FP dataset, whereas the boundary layer height was simulated by WRF in WRF-GC. Compared to the observations, the PBLH in the GEOS-FP dataset were generally biased low over eastern China and biased high over the mountainous areas in southwestern China and western China. This likely was a major reason for the severe overestimation of surface PM_2.5 concentrations in the GEOS-Chem Classic nested-China simulation over eastern China. In comparison, the WRF-GC model correctly represented the PBLH over most regions in China, which was critical to its more accurate simulation of surface PM_2.5 concentrations.

Figure 5 Comparison of the simulated (fill contours) and observed (fill symbols) planetary boundary layer heights (PBLHs) at 08:00 LT (a, b) and 20:00 LT (c, d) averaged between 22 and 27 January 2015. (a, c) PBLH from the GEOS-FP dataset, which was used to drive the GEOS-Chem Classic nested-China simulation, and (b, d) PBLH simulated by the WRF-GC model.

5 Computational performance and scalability of WRF-GC

5.1 Computational performance of the WRF-GC model

We evaluated the computational performance of a WRF-GC simulation and compared it with that of the GEOS-Chem Classic nested-grid simulation of a similar configuration. We configured the WRF-GC and GEOS-Chem Classic nested-grid simulations for the exact same domain (as shown in Fig. 2a), with the exact same projection and horizontal resolution (0.25^∘ latitude $\times \mathrm{0.3125}{}^{\circ }$ longitude resolution, 225×161 atmospheric columns). The GEOS-Chem Classic nested-grid simulation had 47 vertical levels, and the WRF-GC simulation comparably had 50 levels. Both simulations used the same emissions and chemical configurations with a total of 241 chemical species and 10 min external chemical time steps. WRF-GC calculated meteorology online with a 2 min dynamic time step. The GEOS-Chem Classic nested-grid simulation calculated transport (5 min time step) driven by archived GEOS-FP meteorological data read from an ethernet-connected hard disk array. WRF-GC used MPI parallelization, while GEOS-Chem Classic used OpenMP. Both simulations were conducted for 48 h with scheduled output for every 1 h. Both simulations were executed on the same single-node hardware with 32 Intel Broadwell physical cores and an ethernet-connected hard disk array.

Figure 6 compares the computational wall times for the WRF-GC and the GEOS-Chem Classic nested-grid simulations. The total wall time for the WRF-GC simulation was 5127 s, only 31 % of the wall time for the GEOS-Chem Classic nested-grid simulation (16 391 s). We found that the difference in computational efficiency was mainly due to the much faster dynamic and transport calculations in the WRF-GC model relative to the transport calculation in the GEOS-Chem Classic nested-grid model. In WRF-GC, the wall time taken up by the entire WRF (including transport, physics, I/O, and model initiation) was 2462.5 s. In the GEOS-Chem Classic nested-grid simulation, 50 % (8192 s) of the total wall time was used for the transport of tracers, including large-scale advection (6355.7 s), convective transport (694.2 s), and boundary layer mixing (1142.5 s). As a CTM, the GEOS-Chem Classic read archived meteorological data for the entire domain at 3-model-hour intervals from hard drives using a single computational core, which becomes increasingly burdensome for simulations with more grid boxes. In comparison, WRF-GC calculated meteorology online in node memory and updated the model boundary conditions from hard drives every 6 model hours. Finally, the OpenMP parallelization in GEOS-Chem Classic was only at the loop level. In contrast, WRF-GC used MPI parallelization to decompose domain at the model level, thus parallelizing all code within the GEOS-Chem module. As a result, the same chemistry routines actually ran faster in WRF-GC than they did in GEOS-Chem Classic. In WRF-GC, the chemistry routines (deposition, emissions computation, convection, boundary layer mixing, and chemistry) took up 2462.5 s in total; the same routines took up 5263.5 s in GEOS-Chem Classic. The WRF-GC coupler consumed negligible wall time (39 s) in this test simulation.

Figure 6 Comparison of wall time for the WRF-GC model (v1.0) and the GEOS-Chem Classic nested-grid model (v12.2.1).

A side-by-side wall time comparison between WRF-GC and WRF-Chem is difficult to do, because (1) the chemical routines in the two models are very different, and (2) WRF-Chem has many possible configurations for chemistry. Nevertheless, we conducted one test simulation with WRF-Chem using a typical chemistry option (CBMZ-MOSAIC gas-phase chemistry, four-bin aerosol microphysics and chemistry, and aqueous reactions; chem_opt = 9 in registry.chem) with a total of 133 chemical species. This WRF-Chem simulation was configured for the same domain, at the same horizontal and vertical resolutions, and used the same physical and dynamical options as those in the WRF-GC simulation described above. The total wall time for the WRF-Chem simulation was 9985.8 s, which was almost twice as long as the wall time of WRF-GC (5127 s). Chemical routines in this WRF-Chem test simulation took up 61 % of the total wall time (6134.3 s), despite the WRF-Chem having much fewer chemical species than the WRF-GC (241 chemical species). This may be partially due to the computationally intensive bin-resolved aerosol microphysics calculation in the WRF-Chem simulation.

5.2 Scalability of the WRF-GC model

We analyzed the scalability of the WRF-GC model using the 48 h simulation described in Sect. 5.1. The model was compiled using the Intel C and Fortran compilers (v16.0.3) and the mvapich2 (v2.3) MPI library. The computing environment (Tianhe-1A) had 28 Intel Broadwell physical cores with 125 GB of RAM per node. Input and output used a networked Lustre high-performance file system.

Figure 7 WRF-GC model scalability by processes. Grey lines indicate perfect scalability, i.e. halved computational time for each doubling of processor cores.

Figure 7 shows the scalability of the WRF-GC simulation in terms of the total WRF-GC wall time, as well as the wall times of its three components: (1) the WRF model (including input/output), (2) the GEOS-Chem model, and (3) the WRF-GC coupler. For the domain of this test simulation, the total wall time and the WRF wall time both scaled well up to 136 cores. This was because the simulation domain was too fragmented above 136 cores, such that the MPI communication time took up much of the runtime, resulting in performance degradation. Chemical calculations in the GEOS-Chem model were perfectly scalable up to the largest number of cores tested (250 cores), consistent with previous GCHP performance analyses (Eastham et al., 2018). Figure 7 also shows that the WRF-GC coupler scaled nearly perfectly and consumed less than 1 % of the total WRF-GC wall time. At above 200 cores there was a slight degradation of the scalability due to cross-core communications at the subdomain boundaries. However, the degradation had negligible impact on the total WRF-GC wall time as the WRF-GC coupler was computationally efficient.

WRF-GC also scales to massively parallel architectures and can be deployed on the cloud, because both the WRF and GEOS-Chem model are already operational on the cloud with their input data readily available (Hacker et al., 2017; Zhuang et al., 2019). We conducted a test simulation running WRF-GC on the Amazon Web Services (AWS) cloud with up to 128 nodes and 4608 cores. The simulation domain was over the continental US at 5 km×5 km resolution (950×650 atmospheric columns), with a 10 s dynamical time step and 5 min chemical time step. The scalability test results are shown in Fig. S3. In this massively parallel environment, WRF-GC scaled well up to 1728 cores, with the chemical module scaling well up to 2304 cores. The WRF-GC coupler took less than 0.2 % of the total computational time in this simulation and scaled perfectly up to 4608 cores. The deployability of WRF-GC on the cloud will enhance WRF-GC's accessibility to new users by saving them the investment in hardware purchases and the effort in downloading and hosting large input datasets locally.

6 Conclusions

We developed the WRF-GC model, which is an online coupling of the WRF meteorological model and the GEOS-Chem chemical model, to simulate regional atmospheric chemistry at high resolution, with high computational efficiency, and underpinned by the latest scientific understanding of atmospheric processes. By design, the WRF-GC model is structured to work with native copies of the parent models and involves no hard-wired code to either parent model. This allows the WRF-GC model to integrate future updates of either parent model with relative ease.

WRF-GC provides current users of WRF-Chem and other regional models with access to GEOS-Chem, which is state of the science, well documented, traceable, benchmarked, actively developed by a large international community, and centrally managed. At the same time, WRF-GC enables GEOS-Chem users to perform high-resolution regional chemistry simulations in both forecast and hindcast mode at any location and time of interest, with high performance.

Our first application showed that the WRF-GC model was able to reproduce the spatiotemporal variation of surface PM_2.5 concentrations over China in January 2015, with smaller biases compared to the results of the GEOS-Chem Classic nested-China simulation. This was partially because the WRF-GC model better represented the pollution meteorology, including the variability of the planetary boundary layer heights, over the region. In addition, the WRF-GC simulation was 3 times faster than a comparable GEOS-Chem Classic nested-grid simulation.

WRF-GC demonstrated good scalability to massively parallel architectures, with near-perfect scalability of its chemistry component. This enables the WRF-GC model to be used on multiple-node systems or high-performance cloud computing platforms, which is not possible with the GEOS-Chem Classic. The GCHP model also scales to massively parallel architectures (Zhuang et al., 2020), but GCHP can only operate as a global model. The deployability of WRF-GC on the cloud will enhance WRF-GC's accessibility to new users.

The WRF-GC coupling structure, including the GEOS-Chem column interface and the state conversion module, are extensible and can be adapted to models other than WRF. This opens up possibilities of coupling GEOS-Chem to other weather and Earth system models in an online, modular manner. Using native, out-of-the-box copies of parent models in coupled models reduces maintenance and avoids branching of the parent model code. It also enables the community to more easily transfer developments in the parent models to the coupled model, and vice versa.

The WRF-GC model is free and open source to all users. The one-way coupled version of WRF-GC (v1.0) is now publicly available at https://wrf.geos-chem.org (last access: 10 July 2020). A two-way coupled version with chemical feedbacks from GEOS-Chem to WRF is under development and will be presented in a forthcoming paper. Further development of WRF-GC will aim to enable nested-domain simulations, support size-dependent aerosol microphysical calculations, as well as further improve the physical compatibility with WRF. We envision WRF-GC to become a powerful tool for research, forecast, and regulatory applications of regional atmospheric chemistry and air quality.

Appendix A: Abbreviations

Abbreviation	Description
ARW	Advanced Research WRF (dynamical core)
BC	Boundary condition
CCN	Cloud condensation nuclei
CTM	Chemical transport model
EC	Elemental carbon
ESMF	Earth System Modeling Framework
GCC	GEOS-Chem Classic
GCHP	GEOS-Chem High Performance
GCM	General circulation model
GDAS	Global Data Assimilation System
GEOS	Goddard Earth Observing System
GEOS-FP	GEOS Forward Processing
GMAO	NASA Global Modeling and Assimilation Office
HEMCO	Harvard-NASA Emissions Component
IC	Initial condition
KPP	Kinetic PreProcessor
MAPL	Model Analysis and Prediction Layer
MERRA-2	Modern-Era Retrospective analysis for Research and Applications, Version 2
MMM	Mesoscale and Microscale Meteorology Laboratory, NCAR
MPI	Message Passing Interface
NCAR	National Center of Atmospheric Research
NCEP	National Centers for Environmental Prediction
NWP	Numerical weather prediction
PBLH	Planetary boundary layer height
POC	Primary organic carbon
SOA	Secondary organic aerosol
WRF	Weather Research and Forecasting model
WRF-Chem	Weather Research and Forecasting model coupled with chemistry
UCX	Unified Chemistry Extension
VBS	Volatility basis set