MPAS is a modeling framework developed to simulate geophysical fluid dynamics over a wide range of scales . Currently four models based on the MPAS framework exist: atmosphere, ocean, sea ice, and land ice . The atmosphere version (MPAS-Atmosphere) solves the compressible, non-hydrostatic momentum and mass-conservation equations coupled to a thermodynamic energy equation . The novel characteristic of the MPAS framework is a C-grid finite-volume scheme developed for a hexagonal, unstructured grid called Spherical Centroidal Voronoi Tessellations (SCVT; ), accompanied by a new scalar transport scheme by . The SCVT mesh can be constructed to have either quasi-uniform grid cell sizes or variable ones with smooth transitions between the coarse- and fine-resolution regions . The C-grid staggering provides an advantage in resolving divergent flows important to mesoscale features, and the finite-volume formulation guarantees a local conservative property for prognostic variables of the dynamical core . MPAS-Atmosphere is available as a stand-alone global atmosphere model with its own suite of sub-grid parameterizations , but we use the MPAS-Atmosphere numerical solver as the dynamical core coupled to the CAM physics parameterizations here. Previous studies using VR meshes have demonstrated that the MPAS dynamical core is able to simulate atmospheric flow across coarse- and fine-resolution regions without unphysical signals . This capability of regional mesh refinement is the main feature that we aim to test in the context of dynamical downscaling for regional climate projections.

The model code base used for our simulations is a beta version of CESM2 (CESM1.5), the same code used by , who focused on the regional refinement capability of the spectral element dynamical core. The atmospheric component model CAM has multiple versions of the physics parameterization package. We use the CAM version 5.4, which is an interim version toward CAM version 6 . The CAM5.4 physics is the default option for CAM in CESM1.5. The parameterization components in CAM5.4 are summarized in Table 1. Their characteristics are documented in detail by , and a variety of diagnostic plots are publicly available . A major difference between CAM5.4 and the previous version CAM5.0 is the prognostic mass and number concentrations of rain and snow in the new cloud microphysics scheme, MG2 . Prognostic concentrations of precipitating particles make the model more appropriate for high-resolution simulations by removing assumptions necessary for a diagnostic approach (e.g., neglecting the advection of precipitating particles; ). The prognostic aerosol scheme is also revised as the four-mode version of the Modal Aerosol Module (MAM4; ), but we only use the diagnostic aerosol scheme for the simulations documented in this paper. Specifically, the monthly mean aerosol mass concentrations for the year 2000 are derived from a previous simulation using CAM version 4 with the prognostic three-moment MAM on a 1∘ grid. Given the prescribed aerosol mass concentrations, aerosol number concentrations are calculated by an empirical relationship between the two concentrations and are then passed to the cloud microphysics.

An early effort to port the MPAS dynamical core to the CESM/CAM model started in 2011 under the “Development of Frameworks for Robust Regional Climate Modeling” project . The hydrostatic solver of the pre-released version of MPAS was coupled to CAM4 by the collaborative work among Los Alamos National Laboratory, Lawrence Livermore National Laboratory, and the National Center for Atmospheric Research. This CAM–MPAS model was extensively evaluated through a hierarchy of experiments . Those studies demonstrated the ability of VR simulations to reproduce the uniform, globally high-resolution simulations inside the refined domain in terms of the characteristics of atmospheric circulations as well as the sensitivity of the physics parameterizations to horizontal resolution. In the idealized aquaplanet configuration with the older CAM4 physics, the resolution sensitivity of moist physics leads to unphysical upscale effects , but these artifacts are mostly muted when an interactive land model is coupled, along with the presence of other forcing such as topography and land–ocean contrast . The non-hydrostatic version of the MPAS dynamical core (the released version 2) was later coupled to CESM version 1.5 to understand the behavior of the CAM5 physics under a wide range of resolutions over seasonal or longer timescales . used this model with a convection-permitting VR mesh (4–32 km) to study the sensitivity of extreme precipitation to several parameters in the CAM5 physics, demonstrating stable coupling between the non-hydrostatic MPAS dynamical core and the global model physics package CAM5 at kilometer-scale resolution. The CAM–MPAS model for the present work is similar to the one used by , except that MPAS v2 is replaced by a more recent version (version 4). The same CAM–MPAS version employed in this study has demonstrated robust performance in simulating the Asian monsoon system using a 30–120 km VR mesh .

The CAM–MPAS coupling is illustrated in Fig.  along with the process-coupling sequence in the host model CESM1.5. The coupling between the non-hydrostatic MPAS and the main driver of CAM uses a Fortran interface and calling sequence similar to the default finite-volume (FV) core and other dynamical cores available in CAM . With this coupling approach, the dynamical core can be switched from the default FV to MPAS core by simply providing a flag “CAM_DYCORE=mpas” to the CESM build script (env_build.xml), along with an appropriate name of the horizontal grid (e.g., “mp120a” has been defined for the UR120 grid following ). The vertical grid in CAM–MPAS follows the height-based coordinate used by MPAS-Atmosphere , but the number of layers (32) and the height of the interface levels are configured to closely match those of the hybrid σ-p coordinate used by other CAM dynamical cores.

The CESM coupler is responsible for time step management and sequential coupling of component models (Fig. ). When CAM is called by the coupler, the CAM driver cycles the dynamics, physics parameterizations, and communication with the coupler. When the dynamics is called by the CAM driver, the MPAS dynamical core receives tendencies of horizontal momentum, temperature, and mixing ratios that are predicted by physics parameterizations and the other CESM component models and that are summed by the CAM driver prior to the communication with MPAS. MPAS cycles its time steps from the previous atmospheric state with the physics tendencies used as forcing terms. After MPAS completes its (sub) time steps, the updated atmospheric and tracer states are passed to CAM through the interface, including hydrostatic pressure, pressure thickness of each grid box, and geopotential height. The last three variables are required by the CAM physics that operates on a vertical column under hydrostatic balance, without the need to know that the vertical column is discretized in a height-based or hybrid pressure-based coordinate. No vertical interpolation nor extrapolation is performed in coupling CAM and MPAS. The CAM–MPAS interface layer also calculates hydrostatic pressure velocity and performs other required conversions (e.g., converts the prognostic winds normal to cell edges to conventional u and v winds at cell centers and converts mixing ratios defined with dry air in MPAS to those with moist air in CAM). Note that the pressure vertical velocity ω passed from MPAS to the CAM driver is diagnosed under the hydrostatic balance and is different from the non-hydrostatic vertical velocity prognostically simulated in the MPAS dynamical core.

A second-order diffusion is added to the top three model layers to produce the so-called “sponge layers” following other CAM dynamical cores . The top model level is located at about 45 km above sea level. This model top is higher than those typically used in MPAS-Atmosphere (≈30 km). On the other hand, the number of vertical levels in CAM5.4 is smaller than the default number of vertical levels in the MPAS-Atmosphere (41 in version 4), resulting in a relatively coarse vertical resolution for a mesoscale model. However, its vertical resolution is within the range used by regional models participating in NA-CORDEX (18–58 levels across models).

This experimental version of CAM–MPAS is available from our private repository on GitHub (see the “Code and data availability” section), but it is not an official release and does not offer the same technical support as other CAM versions. Some model structural differences between CAM and MPAS, such as the vertical coordinate, require further work to improve physical consistency throughout the coupling processes. An ongoing effort to port MPAS to CAM/CESM addresses those remaining technical issues as part of the System for Integrated Modeling of the Atmosphere (SIMA) project .

Three VR grids (50–200, 25–100, and 12–46 km) and two UR grids (240 and 120 km) are used for the CAM–MPAS downscaling experiment (Table ). Figure  illustrates the UR and VR grids and the distributions of grid cell spacing in the three VR grids. The five CAM–MPAS model resolutions are named UR240, UR120, VR50-200, VR25-100, and VR12-46. UR240 has a similar grid spacing to the Max Planck Institute Earth System Model low-resolution version (MPI-ESM-LR; ), whose ocean and sea-ice output is used as boundary forcing for the future experiment (see below). The UR120 grid has a comparable resolution to those of the majority of CMIP5 and CMIP6 models. Although their grid spacing does not exactly match those of the coarse-resolution domains on the VR grids nor the MPI-ESM-LR model, these two UR meshes are readily available from the MPAS website and serve as a reference for the VR simulations. The two VR grids, VR50-200 and VR25-100, are created for this project because similar VR grids were not available from the MPAS mesh archive when the project started. The two meshes are designed to have a rectangular-shaped high-resolution domain over CONUS (Fig. ), resembling the regional model domain for NA-CORDEX . The 12–46 km VR mesh is obtained from the MPAS mesh archive and has a circular and slightly smaller (by ≈30%) high-resolution domain than the other two VR grids but still covers the most of North America (Fig. f).

As the default parameters in the CAM5.4 physics are tuned for the prognostic MAM4 model, we retuned CAM5.4 with the prescribed aerosol and the CAM default FV dynamical core on its nominal 1∘ global grid. No attempt has been made to tune model parameters differently for the MPAS dynamical core nor at each resolution. While we are aware that resolution-dependent tuning and/or scale-aware physics schemes are necessary to fully take advantage of increased resolution , tuning each resolution for both global and regional climate requires extensive effort e.g., and is left for future work. We also note that resolution-dependent tuning is not usually done for the limited-area models that participated in NA-CORDEX and HyperFACETS nor in other coordinated projects that cover multiple model resolutions e.g.,. The following parameters, however, are changed for each resolution: time step lengths, numerical diffusion coefficients, and the convective timescale used in the Zhang–McFarlane (ZM) deep-convection scheme (Table ). In VR simulations, the dynamics time step is constrained by the smallest grid spacing in the refined region (Δx). The dynamics time steps are initially set as Δt=6×Δx and further adjusted to avoid numerical instabilities that tend to occur within the stratospheric jet over the Andes. The physics time step is scaled from the default 1800 s for ≈1∘ grid spacing using the same ratio as grid spacing changes. The convection timescale is then adjusted to scale with the physics time step in order to reduce sensitivities to horizontal resolution and time step .

For all of our simulations, we use a predefined CESM component set “FAMIPC5” that automatically configures CESM and its input data (e.g., trace gas concentrations) following the protocol of the Atmosphere Model Intercomparison Project (AMIP; ). In this configuration, the atmosphere and land models are active, whereas the sea surface temperature (SST) and sea-ice cover fraction (SIC) are prescribed. The River Transfer Model (RTM) is also active to collect terrestrial runoff into streamflow , but it serves only for a diagnostic propose because the ocean model is not active. The so-called “data ocean” model reads, interpolates in time and space, and passes the input SST to the CESM coupler, which calculates fluxes between the atmosphere and ocean . The Community Ice Code version 4 (CICE4) is run as a partially prognostic model by reading prescribed sea-ice coverage and atmospheric forcing from the coupler to calculate ice–ocean and ice–atmosphere fluxes .

The land component is the Community Land Model version 4 (CLM4; ), which simulates vertical exchanges of energy, water, and tracers from the subsurface soil to the atmospheric surface layer. CLM4 takes a hierarchy-tiling approach to represent unresolved surface heterogeneities, distinguishing physical characteristics among different surface land covers (e.g., vegetated, wetland, lake, and urban), soil texture, and vegetation types . While CLM4 is able to simulate the carbon and nitrogen cycles and transient land cover types, these biogeochemical functionalities are turned off. Instead, our simulations use a prescribed vegetation state (leaf area index, stem area index, fractional cover, and vegetation height) that roughly represents the conditions around the year 2000 based on remotely sensed products . The land cover types are also prescribed as the conditions around the year 2000 and are fixed throughout the simulations in both the eval and rcp85 experiments. These land surface settings are again consistent with the models that participated in NA-CORDEX. Note that the spatial resolution of the original data to derive CLM's land surface characteristics varies from 1 km to 1.0∘, with 0.5∘ being considered as the base resolution (Oleson et al., 2010). These input data are available from the CESM data repository .

The process coupling in CESM has already been illustrated in the previous section (Sect. , Fig. ). In our experiment, the CLM4 land model, data ocean, and CICE4 sea-ice model are configured to run on the MPAS horizontal grid. This way, the state and flux data between different model components do not need to be horizontally interpolated during the model integration. The RTM model in a diagnostic mode runs on its own 0.5∘ grid. The data ocean and RTM communicate with the coupler once and eight times per day, respectively, while CAM, CLM4, and CICE4 run and communicate through the coupler at the same time step.

The experiment is composed of decadal simulations for the present day and the end of the 21st century under the Representative Concentration Pathway (RCP) 8.5, featuring a business-as-usual scenario leading to a radiative forcing of 8.5 W m-2 by the end of this century. The two simulations are named following the CORDEX project protocol: “eval” denotes the historical simulations using reanalysis data for boundary conditions for its principal role of model evaluation against observations; “rcp85” denotes the future simulations in which the external forcings follow the RCP8.5 scenario and the ocean and sea-ice boundary conditions are prescribed by adding the global climate model (GCM)-simulated climate change signals to the historical observations, the so-called pseudo-global-warming experiment. We selected the MPI-ESM-LR model from the eight GCMs considered in NA-CORDEX based on its good performance with respect to the warm-season precipitation over the western and central US .

All of the input data required to reproduce our simulations are publicly available (see the “Code and data availability” section). The SST and SIC for the eval run are taken from the ERA-Interim reanalysis . The 6 h ERA-Interim SST and SIC data are averaged to daily values and provided to the model as input; they are then bilinearly interpolated to the MPAS grids by the CESM coupler during model integration. Other model input data include surface topography, initial conditions, and remapping weights between different input data and model grids (Appendix ). All of the surface-related input data are remapped to each MPAS grid prior to the simulations following the CESM1.2 and CLM4 user guide . A set of high-level scripts is now available to help prepare input data for the FAMIPC5 and other similar CESM experiments . Topography input is generated by the stand-alone MPAS-Atmosphere code (init_atmosphere; ), which uses the GTOPO global 30s topography data as the input. The sub-grid topography information required by the gravity wave drag and turbulent mountain stress parameterizations in CAM5.4 are produced using the NCAR_Topo tool by .

As stated above, the future simulation is conducted using the pseudo-global-warming approach e.g., based on the climate change signal simulated by the MPI-ESM-LR model from the CMIP5 archive. Specifically, annual cycles of the daily climatological SST and SIC are obtained from the historical and RCP8.5 simulations of the MPI-ESM-LR model (ensemble member id r1i1p1), and differences between the two periods are calculated for each day of the year and each grid point. This daily climatological difference (ΔSST and ΔSIC) is then added to the SST and SIC from the ERA-Interim data and is prescribed to the model. Other external forcings of solar irradiance, greenhouse gas, ozone, and other tracer gas concentrations are the same as the CESM1.2 RCP8.5 simulation conducted for CMIP5, except for the prescribed aerosol concentrations and land cover characteristics being kept the same as the eval simulation.

The annual average ΔSST and ΔSIC are shown in Fig. e and f, respectively. While the SST and SIC distributions in the present-day period are reasonably simulated by MPI-ESM-LR, regional biases exist over the Southern Ocean, North Atlantic, and off the west coasts of North and South America and South Africa (Fig. a–d). Because ΔSST and ΔSIC are added onto the climatology from ERA-Interim, the future SST and SIC forcings given to CAM–MPAS are different from those in the MPI-ESM-LR model over the biased regions. In Sect.  and Appendix , we briefly compare the CAM–MPAS historical climate and its response to the external forcings with those of the MPI-ESM-LR model. It is shown that, while the base state climate differs between the two models, their changes into the future are rather similar under the same ΔSST and ΔSIC. Also of note is that the SST or near-surface air temperature (TAS) biases and their changes in the MPI-ESM-LR simulations differ from those of fully coupled CESM simulations with CAM5 or CAM6 . Specifically, our CAM–MPAS downscaling data describe the response of the atmosphere to the ocean conditions derived from the external data (as is the case for regional model simulations in NA-CORDEX), which may be very different from the climate evolution simulated by CAM–MPAS being coupled to an active ocean model. Because CAM–MPAS and other VR atmosphere models are typically a part of global coupled climate models, it is possible to run a fully coupled VR simulation, which provide climate change signals that have co-evolved with the same atmosphere model.

Limited-area models participating in NA-CORDEX include another historical simulation called “hist”, in which the lateral and bottom boundary conditions are provided by the driving global models. Also the rcp85 simulations in NA-CORDEX use GCM output directly for boundary conditions (“direct downscaling”), in contrast to adding the climate change signals to the observed present-day boundary conditions. We do not conduct the hist experiment with CAM–MPAS, as our principal goal is to assess the credibility of dynamically downscaled climate by the CAM–MPAS atmosphere model in comparison to observational and other downscaled data, which, at a minimum, requires (1) the eval run with the prescribed ocean boundary conditions from observations, isolating the CAM–MPAS model's bias without the influence of the GCM's SST and sea-ice biases, and (2) the model response to external forcings associated with global warming, which can be reasonably assessed by the pseudo-global-warming experiment (see the general agreement in the large-scale climate response between the CAM–MPAS and MPI-ESM-LR models in Sect. ). An advantage of the global VR simulation in pseudo-global-warming approach is that, unlike adding the mean atmospheric climate change signals to the lateral boundary conditions for regional models, a global VR simulation does include variability and high-order atmospheric responses to the warming. Because our dataset does not have the hist experiment, we will use the terms “eval”, “historical”, and “present-day” interchangeably to refer to the eval simulations.

The atmospheric initial condition for the eval experiment is taken from the ERA-Interim data on 1 January 1989 at 00:00 UTC for all simulations except for the VR12-46 simulation that used data from 1 January 2000 at 00:00 UTC. The land initial condition is taken from the output valid for 1 January 2000 at 00:00 UTC from a 0.5∘ fully coupled CCSM4 simulation for the historical period . The CLM4 land state on the 0.5∘ grid is remapped to the MPAS grids following . Starting from these initial conditions, the model is run for 1 year to spin up the eval simulations. For the future rcp8.5 experiments, the initial condition for each resolution is taken from the 1 January 2011 state of the corresponding eval simulation, followed by 2 years of spin-up simulations. We found that these spin-up lengths are sufficient for the CONUS domain, but they are not necessarily adequate in the deep soil layer for the global domain, particularly at high latitudes (as will be discussed in Sect. ).

To facilitate comparison with other regional models in the NA-CORDEX model archive, the model output on MPAS's unstructured mesh is remapped to a standard latitude–longitude regional grid defined by the NA-CORDEX project (the so-called NAM grid; Fig.  and Table ). Variable names and units used in CAM/CESM are converted to those of NetCDF Climate and Forecast (CF) Metadata Conventions (version 1.6) that are used by NA-CORDEX. Three-dimensional atmospheric variables defined on the terrain-following model coordinate are vertically interpolated to the NA-CORDEX-requested pressure levels (200, 500, and 850 hPa). The following describes how such post-processing was performed.

We mainly used the Earth System Modeling Framework (ESMF) library through the NCAR Command Language (NCL) for regridding MPAS output. The ESMF library provides several remapping methods, among which the first-order conserve method is used for extensive variables and fluxes, and the patch recovery method is used for all other variables. For variables required at a specified pressure level, we first linearly interpolate from the model height level to the pressure level, followed by horizontal remapping. The order of the vertical vs. horizontal interpolation is not expected to be important for the accuracy of subsequent analyses . Note that the three pressure levels available in the post-processed archive are not sufficient to close budget equations of vertically integrated quantities such as moisture and energy (Bryce Harrop, unpublished result). For moisture budget analyses, data users are encouraged to use the vertically integrated moisture fluxes and water vapor path available in the daily variables (Appendix ). For other variables, it is possible to retrieve them at more pressure levels from the monthly or 6-hourly raw model output (Appendix ).

Missing values exist in some variables in the raw model output on the MPAS grids, e.g., soil moisture in the grid points where 100 % of the grid point area is covered by ocean, lake, or glacier. The locations of such missing values do not change with time, and the corresponding grid points are masked when generating regridding weights. Time-varying missing values arise during vertical interpolation to a pressure level over the areas where surface topography crosses the target pressure level. We followed the guidance provided by the NCL website to regrid such time-varying missing values . Specifically, we first remap a binary field defined on the source MPAS grid: all values are one where the vertically interpolated pressure-level variable is missing, and they are zero everywhere else. By remapping such a field from the original MPAS grid to the destination grid, we can identify, using nonzero values, which destination grid points are affected by the missing values on the original grid, and the remapped pressure-level variables in these destination grid boxes are set missing. This rather cumbersome procedure can be replaced by a remapping utility recently enhanced in the NetCDF Operator (NCO; ).

NA-CORDEX documents minor artifacts due to interpolation by the patch recovery method (e.g., small negative values for nonnegative variables such as relative humidity) . The influence of horizontal regridding, or interpolation, on the statistics has also been noted by previous studies . To understand the effect of regridding in our post-processing, Fig.  compares selected statistics calculated on the original and remapped daily precipitation using different remapping methods, bilinear, patch recovery, first-order conserve, and second-order conserve, available from the ESMF library . The regridding effect on the (spatial) mean is negligibly small using any of the regridding methods. As shown in Fig. a, the global annual mean precipitation (3.004 mm d-1) is nearly identical (to the accuracy of 10-3 mm d-1) for the original and the regular 1∘ latitude–longitude grid after remapping.

The variance loss due to remapping is typically ≈6 %–8 % for daily precipitation. The magnitude of variance loss depends on which variable is remapped – a variable with a smoother spatial structure than precipitation (e.g., atmospheric temperature) is less affected by regridding. At the global scale, ≈6 %–8 % loss of variance can be larger than year-to-year sampling variability, as illustrated in Fig. b. The second-order conservation method retains the spatial variance slightly better than the other methods. At regional scales, sampling uncertainty from different time periods (each sample is 5 years long here) can be as large as the smoothing effect. This is illustrated in Table  (from the third to sixth rows) based on the statistics of daily precipitation in the CONUS sub-domain east of the Rockies, calculated on the original VR25-100 grid and conservatively remapped to the NAM-22i grid. We avoid the Rockies and other mountainous regions where year-to-year variability is so large that our sample size is not long enough to reliably estimate spatial variances. The third and fourth rows present the statistics on these two grids from the years 2001 to 2005, while the fifth and sixth rows are from the years 1991 to 1995 and 1996 to 2000, respectively, on the NAM-22i grid. The 5-year average of the spatial variance is 32.37 mm d-1 on the original grid for 2001–2005, which is reduced to 29.85 mm d-1 after regridding. The spatial variance from the other 5-year period can differ from the variance from 2001 to 2005 by as much as the regridding loss. Similar magnitudes of smoothing effect and sampling uncertainty are also found in kurtosis and extreme values represented by the 95th to 99.99th percentiles. These differences are not visible on the daily precipitation histograms calculated on the original VR25-100 grid, the remapped NAM-22i grid, and the UR120 output on its raw MPAS grid for the same CONUS sub-domain (Fig. c). The two histograms of VR25-100 are visually identical, and the difference from the UR120 precipitation is clearly distinguishable.

Two other points notable in Table  are as follows: (1) the smoothing effect becomes weaker with finer grid resolutions based on the three VR resolutions and (2) successive remapping back from the regional NAM-22i to the original grid (the seventh row) leads to a further loss of the variance and other moments but to a lesser degree compared with the first remapping. Similar smoothing effects from the first and second remapping are observed in the output from the WRF model on a 25 km grid from the NA-CORDEX archive. Despite the fact that WRF uses a regular latitude–longitude grid that is similar to the NAM-22i grid, regridding effects on the selected statistics resemble those on the CAM–MPAS output. For example, regridding VR25-100 output loses ≈8% of the daily precipitation variance by the first remapping, while the 25 km WRF simulation loses 6 %. The histograms of daily precipitation in the WRF 25 km simulation are shown in Fig. d, again confirming that the histograms are not visually affected by regridding. Given such a priori knowledge of the regridding effect and sampling uncertainty at regional scales, we do not expect that the remapping effect would seriously affect the statistical inference of regional climate metrics.

Post-processed monthly and daily variables in the “essential” and “high priority” list of the NA-CORDEX archive are accessible from the Pacific Northwest National Laboratory DataHub. All of the variables and temporal frequencies are available from the NERSC High Performance Storage System (HPSS), made accessible through web browsers by the NERSC Science Gateway Service (see the “Code and data availability” section). All variables requested from the experiment protocol are two-dimensional at a single level. Appendix  lists the post-processed variables.

File names, attributes, and coordinates of the reported variables and their file specification follow the CORDEX archive design and NA-CORDEX data description . The file name is composed of the following elements: [variable name].[scenario].[driver].[model name].[frequency].[grid].[bias correction].[start month]-[end month].[version].nc. In the CAM–MPAS dataset, the scenario is either eval for the historical period or rcp85 for the pseudo-warming future simulation. The driver is “ERA-Int” for the historical period and “ERA-Int-MPI-ESM-LR” for the rcp85 case. Post-processing of the current CAM–MPAS simulations does not involve any bias corrections; hence, it is labeled as “raw”. The major version refers to different production simulations, and the minor version refers to changes/corrections in the post-processing stage. The publicly available CAM–MPAS output is either “v3” or “v3.1”; the major version is 3 because it was necessary to rerun simulations twice due to major changes in model configurations, and the minor revision involves a different treatment of missing values arising from vertical interpolation to a pressure level (see Sect. ). With the other straightforward file name elements, an example file name for a daily precipitation data in the historical run of CAM–MPAS VR50-200 reads as follows: pr.eval.ERA-Int.cam54-mpas4.day.NAM-44i.raw.198901-201012.v3.nc. In contrast, an example file name for a daily precipitation data in the future pseudo-warming reads as follows: pr.rcp85.ERA-Int-MPI-ESM-LR.cam54-mpas4.day.NAM-44i.raw.207901-210012.v3.nc.

Raw CAM–MPAS output on the global MPAS grid (i.e., not remapped to a regional latitude–longitude grid) is also available from the NERSC HPSS space. Appendix  provides more information about the MPAS unstructured mesh, links to the archive directory, and other resources to help analyze the raw MPAS data. The NERSC data archive also contains example scripts and variables necessary to process model variables on the MPAS grid (e.g., latitude and longitude arrays).

In this section, we discuss some computational aspects of our simulations, as one of the motivations to use a global VR framework is its computational advantage compared with a global high-resolution simulation. On the other hand, global VR simulations are expected to be more expensive than limited-area model simulations if the cost for the host GCM simulations that provide boundary conditions is not considered. For example, the VR grids used in this study have 1.1–2.6 times more grid columns than the limited-area grids used by the RegCM4 and WRF models in the NA-CORDEX and HyperFACETS archives (Tables , ). Here, we do not compare the simulation cost of the CAM–MPAS VR configurations against regional models but instead focus on how the cost of CAM–MPAS simulations differs between the UR and VR grids and between the lower and higher resolutions.

All of our simulations were run at NERSC. The following result is obtained from the production simulations and not a systematic scaling analysis of the CAM–MPAS code nor NERSC systems. The system configurations (e.g., number of nodes) of our production simulations are not only based on good throughput but also on simulation cost as well as expected queue wait time (Fig. ), which often accounts for the majority of the total production time (e.g., the average queue wait time for VR25-100 is approximately 3 times the actual computing time). All simulations used only the distributed-memory Message Passing Interface (MPI) parallelism, i.e., shared-memory parallelism (OpenMP) is not used. The main computing system at NERSC switched from Edison to Cori when the production simulations of the CAM–MPAS model were starting . The newer system Cori is partitioned into two subsystems, Cori Haswell (HW) and Cori Knights Landing (KNL). As discussed below, the CESM–CAM–MPAS code showed large differences in performance on KNL and other systems, posing a significant impact on our production cost. Interested readers are referred to Appendix  for further details of our runtime configurations and the characteristics of the NERSC systems.

Three simulations that are not part of the CAM–MPAS downscaling dataset are also included in the following as references: (1) the default FV dynamical core on the nominal 1∘ grid (FV 1∘), (2) the same model configuration as UR120 but using the newer version of the CAM–MPAS model that will be released as an official option of CESM2 (UR120-new), and (3) CAM–MPAS on a quasi-uniform 30 km grid (UR30). These three simulations were run for other projects but with a similar set of file output (monthly, daily, 6 h, 3 h, and hourly output) for more than 5 years. All simulations use the same CAM5.4 physics with prescribed aerosol.

Figure a visualizes the simulation cost vs. total MPI tasks used, as often used in cost-scaling studies. Table  lists the numerical values used in the figure. Although scatters in the data from different computing systems are notable, there is a clear trend to which we can fit a curve. The blue line represents a power function (y=axb) fitted to the simulation cost in the log–log space. The exponent b (the slope of a straight line on the log–log plot) is 1.54 with a 95% confidence interval of 0.50, exhibiting a weak but nonlinear increase. The nonlinear increase is expected because linearly increasing cost is only possible for an idealized case, as also shown in the figure. The green line represents an ideal situation that the parallel part of the code speeds up linearly with additional resources (an ideal weak scaling; Eq. 5.14 in ), whose cost thus increases linearly with the number of MPI ranks (slope of 1). The orange line of a constant cost applies only to the case where the size of the problem (e.g., number of grid columns) stays the same so that using more resources shortens the simulation time. This is an ideal “strong scaling” and is not applicable to the cost scaling for different resolutions over a fixed global domain. It is obvious from this comparison that larger resource use for higher resolutions on a fixed domain size, such as the global domain, always increases the computing cost nonlinearly.

There are several reasons for the nonlinear increase in the simulation cost against resources used, such as communication and load imbalance . For estimating the simulation cost of a given MPAS grid, we found that it is simpler to use the number of calculations (physics and dynamics time steps) per simulated day across all of the grid boxes in the global domain: Ncalc= (number of grid columns) × (number of vertical levels) × (number of time steps per day). Plotting simulation cost as a function of Ncalc (Fig. b), the fitted curve exhibits a slope of approximately 1. Looking at Ncalc as a function of the number of grid columns, it appears to be separated into two groups of VRs and URs, indicating the time step constraint from the high-resolution domains in VRs (Fig. c). The least-squares-fitted power functions have exponents of 1.45 for both VR meshes and UR meshes. This weak nonlinearity presumably comes from the dependence of time step length on grid spacing, which then becomes an additional implicit dependence on the numbers of grid columns.

As a specific example of VR vs. UR comparison, we take VR25-100, UR30, and UR120, as the latter two URs have comparable grid spacings in the high- and low-resolution regions of the VR25-100 grid. We use Ncalc of the simulations conducted on KNL to gauge the computational advantage of the VR25-100 against UR30, a uniform high-resolution simulation, as well as the extra cost added by the regional refinement to a uniform low-resolution simulation, UR120. The actual values of Ncalc for these three resolutions are shown in the third column of Table  and suggest UR30 to be 48 times more expensive than UR120 and VR25-100 to be 16 times more costly than UR120. The actual simulation cost closely follows the Ncalc scaling; 1 simulation year of UR30 (480.0×103 NERSC h sim. yr-1) is 48 times more expensive than that of UR120 (10.1×103 NERSC h sim yr-1.) The actual cost of VR25-100 is just 11 times that of UR120, which is lower than that expected from Ncalc, possibly reflecting the error from using an empirical curve fitted to three different systems in the single KNL system. In this case, VR25-100 achieves a factor of 4 computational advantage compared with UR30 for obtaining a similarly high-resolution grid over CONUS.

A couple of other points are noted in Table  and Fig. . First, the computational cost of CAM–MPAS UR120 and the default dynamical core FV 1∘ is comparable (1.9 vs. 2.1 sim. yr d-1 for CAM–MPAS UR120 and CAM–FV 1∘, respectively). Second, the model throughput (cost) of VR12-46 is 0.2 sim. yr d-1, half (double) that of UR30, despite the fact that these two grids have the same number of columns and that the simulations are run with similar numbers of columns per MPI task. The main reason for the difference is likely the shorter time steps (about one-third) in VR12-46 than in UR30 due to the numerical constraint imposed by the smallest grid spacing in the high-resolution domain. Lastly, we get consistently lower throughput and higher cost on Cori KNL than on the other two systems. Our experiment and previous studies suggest a few compounding reasons (Appendix ), such as inefficient memory management for some global arrays, poor vectorization, and less focus on shared-memory parallelism of the CAM5/MPASv4 source code, which are not aligned well with the wider-vector and many-core architecture of KNL. However, the shorter expected queue time on KNL than HW (Fig. ) makes KNL our main system for production. The weaker performance of the experimental CAM–MPAS code on KNL leads to a higher computational cost than our initial estimate for VR12-46, limiting the length of VR12-46 simulations to be half of other simulations. More importantly, the code characteristics described above are not necessarily unique to the CAM5/MPASv4 codes but may be common in other global or regional climate models in which many lines of the codes are written by domain scientists with little attention to code optimizations. Such climate models are not likely to be efficient on emerging, more energy-efficient HPC architecture similar to KNL for having wider vector units and more cores per node (and less memory per core) than previous systems. For example, two new systems being deployed to HPC centers in the United States – Perlmutter to NERSC and Derecho to the NCAR-Wyoming Supercomputing Center – share such characteristics in their CPU nodes.

Fortunately, some of the computational problems with the CAM–MPAS model have been resolved through the MPAS-Atmosphere optimization, ongoing effort to port the later version 6 of MPAS-Atmosphere to CESM2 (the SIMA project), and other numerous changes across the CESM source code from CESM1.5 to CESM2. Those updates lead to an almost 80% speedup of the UR120 throughput, as can be seen from the UR120 and UR120-new simulations in Table . Some of the speedup comes from different compiler optimizations used for the two simulations, but the code development plays a major role in this performance improvement. The Cori system is retiring, but the computational advantage of the new code is expected to be applicable to other systems, including the new NERSC system Perlmutter. We expect that decadal simulations on the VR12-46 grid or even convection-permitting VR meshes will be feasible using the newer CAM–MPAS code or the SIMA atmospheric general circulation model with MPAS as its dynamical core option. Multi-season convection-permitting simulations have been already carried out with the new SIMA-MPAS model .

We briefly review selected aspects of the simulated climate. The focus here is the climate statistics at the global-scale and over the regions outside the VR high-resolution domain of North America. This is because, although the post-processed datasets cover a broad area encompassing the NA-CORDEX domain (Fig. a), the limited-area grid does not allow one to infer remote sources of large-scale forcings and their dependency on model resolution, which may be important to understand processes responsible for projected changes within the high-resolution domain. Appendix  presents additional figures and a table. For the downscaled regional climate, Appendix  provides a general overview of the model performance focusing on the CONUS region. The main finding of the regional assessment is that the performance metrics of precipitation improve with higher resolution, but the results are more mixed for other variables. Moreover, the resolution sensitivity of precipitation becomes weaker within the North American domain compared with the global statistics, which is shown below. A separate, systematic investigation of the regional climate in comparison with other limited-area models is being conducted and will be reported elsewhere.