We initially focus on four geographical regions where liquid clouds occur frequently and extensive measurements are available from ARM and other agencies: eastern North Atlantic (ENA), northeastern Pacific (NEP), central US (CUS, where the ARM Southern Great Plains, SGP, site is located), and Southern Ocean (SO). Aerosol properties also vary among these regions. Six field campaigns from these four test beds are selected in version 1 of ESMAC Diags (Table 1). HI-SCALE and ACE-ENA are based on long-term ARM ground sites with aircraft field campaigns sampling below, within, and above convective and marine boundary layer clouds, respectively, within a few hundred kilometers around the sites. CSET and MAGIC are field campaigns with respective aircraft and ship platforms sampling transects between California and Hawaii, which is an area characterized by a transition between stratocumulus- and trade-cumulus-dominated regions. SOCRATES and MARCUS are field campaigns with respective aircraft and ship platforms based out of Hobart, Australia. Aircraft transects during SOCRATES extended south to around 60∘ S, while ship transects during MARCUS extended southwest from Hobart to Antarctica. The aircraft (black) and ship (red) tracks for these field campaigns are shown in Fig. 3.

The instruments and measurements used in ESMAC Diags version 1 are listed in Table 2. All in situ measurements are converted to under ambient temperature and pressure. Note that some instruments are only available for certain field campaigns or failed operationally during certain periods; thus, model evaluation is limited by the availability of data collected in each field campaign. ARM data usually include quality flags indicating bad or indeterminate data. These flagged data are filtered out, except for surface condensation particle counter (CPC) measurements for HI-SCALE. CPC data flagged as greater than a maximum value (8000 cm-3) are retained, as aerosol loading can be higher than the abovementioned value during NPF events. This exception ensures a reasonable diurnal cycle, as shown in Sect. 3.3. For some data that do not have a quality flag, a simple minimum and maximum threshold is applied (e.g., a 500 cm-3 maximum threshold is used for each Ultra-High Sensitivity Aerosol Spectrometer, UHSAS, bin from the NCAR research flight measurements).

For some field campaigns (HI-SCALE and ACE-ENA), there are several instruments (e.g., fast integrated mobility spectrometer, passive cavity aerosol spectrometer probe, and optical particle counter for aircraft; scanning mobility particle sizer and nano-scanning mobility particle sizer for ground) measuring aerosol size distribution over different size ranges. These datasets are merged to create a more complete size distribution. In ESMAC Diags v1, aerosol “size” refers to the mobility and optical dry diameter of particles. The aerosol concentrations in the “overlapping” bins measured by multiple instruments are weighted by the uncertainty of each instrument based on the knowledge of the ARM instrument mentors. An example of the merged aerosol size distribution and individual measurements for one flight in ACE-ENA is shown in Fig. 4. Ranging from 101 to 104 nm, the merged aerosol size distribution data account for the ultrafine, Aitken, and accumulation modes.

Although these measurements are considered as “truth” when evaluating ESMs, we note that they are subject to limitations and uncertainties due to factors such as theoretical/methodological formulations, sampling representativeness, instrumental accuracy and precision, imperfect calibration, and random errors. In addition, sampling volumes differ between observations and model output and are not reconcilable. It is difficult to quantify every aspect of observational uncertainty within the context of interpreting comparisons with model output, but we try to discuss some of them in this study to the best of our knowledge. Percentiles (either 25th–75th or 5th–95th) are used in some analyses of this study to approximate data variability that is likely to be much higher than measurement uncertainty.

In this study, we run EAMv1 from 2012 to 2018, covering all six field campaign periods introduced previously, with enough time for model spin-up. The model is configured to follow the Atmospheric Model Intercomparison Project (AMIP) protocol (Gates et al., 1999) with real-world forcings (e.g., greenhouse gases, sea surface temperature, and aerosol emissions). For each simulation year, we use the year 2014 emission data from Phase 6 of the Coupled Model Intercomparison Project (CMIP6), as the emission data do not cover years after 2014. The simulated horizontal winds are nudged towards the Modern-Era Retrospective analysis for Research and Applications, Version 2 (MERRA-2, Gelaro et al., 2017) with a relaxation timescale of 6 h. Using such a nudging configuration, previous studies (Sun et al., 2019; Zhang et al., 2014) have shown that the large-scale circulation is well constrained in the nudged simulation, especially for the mid- and high-latitude regions. The simulation uses a horizontal grid spacing of ∼ 1∘ (NE30, the number of elements along a cube face of the E3SM High-Order Methods Modeling Environment, HOMME, dynamics core) with a 30 min time step. We saved hourly output for comparison with field campaign measurements. The diagnostics package post-processes 3-D model variables associated with aerosol concentration, size, composition, optical properties, precursor concentration, CCN concentration, and atmospheric state variables. The size of the output data is reduced by saving 3-D variables only over the field campaign regions. The model configuration and execution scripts are uploaded as an electronic supplement to this paper. Users can apply it in their own E3SM simulations (or output similar variables if running other models) to use this package.

We extracted model output along the aircraft (ship) tracks using an “aircraft simulator” (Fast et al., 2011) strategy to facilitate comparisons of observations and model predictions. At each aircraft (ship) measurement time, we find the nearest model grid cell, output time slice, and vertical level of the aircraft altitude (or the lowest level for ship) to obtain the appropriate model values. As both spatial and temporal mismatch exist between model output and field measurements, the evaluation focuses on overall statistics. We also calculate the aerosol size distribution from 1 to 3000 nm at 1 nm increments from the individual size distribution modes in MAM4 to facilitate comparisons with the observed aerosol number distribution that has different size ranges for different instruments. All of these variables are saved in separate directories according to the specific aircraft (ship) tracks, as indicated in Fig. 2.

Currently, ESMAC Diags produces the following diagnostics and metrics:

mean value, bias, root-mean-square error (RMSE), and correlation of aerosol number concentration;

Aerosol properties are highly dependent on location and season. Figure 5 shows the mean aerosol size distribution for each of the four test bed regions. For HI-SCALE and ACE-ENA, the two IOPs operated in different seasons are shown separately. Table 3 shows the mean aerosol number concentration from these field campaigns for two particle size ranges: >10 and >100 nm. The interquartile range (25th and 75th percentiles) is also shown to illustrate the variability in space and time. Among the four test bed regions, the CUS region has the largest aerosol number concentrations, as the other field campaigns are primarily over open ocean. Overall, EAMv1 overestimates Aitken-mode (10–70 nm) aerosols and underestimates accumulation-mode (70–400 nm) aerosols for the CUS and ENA regions, suggesting that processes related to particle growth or coagulation might be too weak in the model. Over the NEP region, EAMv1 overestimates aerosol number for particle sizes >100 and >10 nm (Table 3), both at the surface and aloft. Over the SO region, which is considered a pristine region with a low aerosol concentration, observations show a significant number of particles <200 nm in both aircraft and ship measurements (Fig. 5). The mean aerosol number concentration over the SO region is comparable to or even greater than the other ocean test beds (Table 3). In contrast, EAMv1 simulates a clean environment with the lowest aerosol number concentrations among the four regions. These types of comparisons demonstrate the need for additional analyses to understand why the SO has a similar aerosol number to other ocean regions and why EAMv1 cannot simulate this feature. The observed 75th percentiles are sometimes smaller than the mean value (Table 3), indicating a skewed aerosol size distribution with a long tail in the large aerosol size. EAMv1 usually produces a smaller interquartile range than the observations, likely because the current model resolution is too coarse to capture the observed spatial variability in aerosol properties.

Both the observed and simulated aerosol size distribution and number concentration show large variability during these field campaigns. Over the period of a few weeks or longer, aerosol number can vary by an order of magnitude between the 10th and 90th percentiles, especially for small particles (Fig. 5). Figure 6 shows the mean aerosol size distributions for two flight days during HI-SCALE: one with a large number of small (<70 nm) particles (14 May) and the other (3 September) with fewer small particles but more accumulation-mode (70–300 nm) particles. On both days, EAMv1 reproduces the observed planetary boundary layer height (PBLH) reasonably well with sufficient samples below and above the planetary boundary layer (PBL). On 14 May, EAMv1 reproduces the observed aerosol size distribution reasonably well, both within the PBL and in the lower free atmosphere. However, on 3 September, EAMv1 produces too many aerosols in the Aitken mode and too few accumulation mode aerosols in the PBL. In the free atmosphere, EAMv1 reproduces the lower concentration of Aitken-mode aerosols but still underestimates the accumulation mode. Such contrasting cases will be useful to help diagnose the specific processes contributing to model uncertainties in future analyses. This large day-to-day variability also indicates that long-term measurements are needed to avoid sampling bias in building robust statistics in aerosol properties. The next version of ESMAC Diags will be extended to include the available long-term ARM measurements at SGP, ENA, and other sites outside of the field campaign time periods.

A research aircraft is the primary platform to provide information on the vertical variations in key aerosol properties that cannot be obtained accurately by remote sensing instrumentation. In this section, we show an example of evaluating vertical profiles of aerosol properties using aircraft measurements as well as illustrating the capability to evaluate multiple model simulations with ESMAC Diags. In addition to the standard EAMv1 simulation described in the previous section, we performed an EAMv1 simulation using the regionally refined mesh (RRM) (Tang et al., 2019). The model is configured to run with a horizontal grid spacing of ∼ 0.25∘ over the continental US and ∼ 1∘ elsewhere. The two model configurations are identical except for the higher spatial resolution (including primary aerosol emissions) in the RRM over the continental US. All aircraft measurements with a cloud detected simultaneously (cloud flag =1) were excluded.

Figure 7 shows vertical percentiles of aerosol number concentration, composition, and CCN number concentration among all of the HI-SCALE aircraft flights. Note that aircraft rarely flew above 3 km during HI-SCALE; thus, the sample size above that altitude is much smaller. The observed aerosol concentrations of number and chemical composition decrease with height, as the major sources of aerosols (anthropogenic, biogenic, and biomass burning) (Liu et al., 2021) are from precursors emitted near the surface and chemical formation within the PBL. EAMv1 generally simulates less variability than observations, except for sulfate. Overall, EAMv1 reproduces the observed mean aerosol number concentration for aerosol size >10 nm but underestimates the number of larger particles >100 nm during HI-SCALE (Table 3). The model also overestimates sulfate and underestimates organic matter concentrations when compared with aircraft AMS measurements. Its underestimation of the CCN number concentration is consistent with the underestimation of aerosol number concentration for diameters >100 nm but contrary to the overestimation of sulfate. A similar relationship is seen for ACE-ENA, which is described later in this section.

The differences in sulfate and organic matter aloft are consistent with the longer-term surface measurement differences shown in Fig. 8, suggesting that this is a model bias. Note that near-surface measurements by aircraft are not always consistent with ground measurements (e.g., total organic matter in IOP1), which reflects the large spatial variability in aerosol properties associated with the aircraft flight paths up to a few hundred kilometers around the ARM site. The greater fraction of sulfate in EAMv1 suggests that the simulated aerosol hygroscopicity is likely higher than observed. Currently only these two species are available in both EAMv1 and AMS/ACSM observations for comparison purposes. Zaveri et al. (2021) recently added chemistry associated with NO3 formation in MAM4, which is expected to be implemented in a future version of EAM.

Ongoing developments in E3SM will soon permit regionally refined meshes with grid spacings as small as ∼ 3 km as well as global convection-permitting simulations (Δ×∼ 3 km); therefore, this diagnostics package is designed to be flexible in scale to take advantage of higher-resolution ESM simulations that are more compatible with high-resolution in situ aerosol observations. This study demonstrates this ability by using a 0.25∘ RRM simulation. Overall, the RRM analyzed here has similar biases as EAMv1, with differences that vary seasonally. The interquartile ranges in Fig. 7 show that the variability in organic aerosols and CCN from the EAMv1 and RRM simulations are similar. However, the variability in sulfate in RRM is larger than EAMv1 and observations during the spring IOP (IOP1). During the summer IOP (IOP2), the variabilities in sulfate in EAMv1, RRM, and observations are similar, and the sulfate concentrations from RRM are closer to observed values than EAMv1. Individual time series from the RRM simulation are still too smooth to capture the fine-scale variability in aerosols in observations (not shown). We expect E3SM to capture more fine-scale variabilities related to urban and point sources of aerosols and their precursors when the simulation grid spacing is further reduced to ∼ 3 km. A sensitivity study will be conducted when this high-resolution version of E3SM simulation becomes available.

Figure 9 shows the vertical variation in percentiles of aerosol properties for ACE-ENA. The observed aerosol number concentrations, composition masses, and CCN number concentrations are much smaller than those for HI-SCALE, representing a cleaner ocean environment. EAMv1 produces larger mean values than the observations for all of these quantities. The overall variabilities in predicted aerosol number and concentrations of sulfate and organic matter are also greater than observed. Note that the observed variabilities for HI-SCALE are much larger than for ACE-ENA, indicating that EAMv1 has smaller location variation in aerosol variabilities. The observed total organic concentration shows a peak aloft between 1.6 and 2.2 km, corresponding to the level of the CCN number concentration peak. This implies that a major source of aerosols or precursors is free tropospheric transport (Zawadowicz et al., 2021). This peak in the total organic concentration aloft is also captured by the model.

The bar plots of aerosol composition at the surface during ACE-ENA from the ACSM instrument and EAMv1 (Fig. 10) illustrate a similar bias in sulfate and organic mass as aloft. While the surface sulfate measurements are like those from the aircraft at the lowest altitudes, the observed surface organic matter is much higher than aloft, particularly during IOP2. The differences in these measurements may be due to local effects or possible contamination from aircraft, as the surface station is located near an airport on an island.

Aerosol number concentrations and size distributions are highly impacted by NPF events (Kulmala et al., 2004), which further influence CCN concentration (e.g., Kuang et al., 2009; Pierce and Adams, 2009) and ultimately cloud properties. NPF and subsequent particle growth are frequently observed in the CUS region (Hodshire et al., 2016). As described by Fast et al. (2019) and shown in Fig. 11a, several NPF events were observed during the HI-SCALE spring IOP (IOP1). Large concentrations of aerosols smaller than 10 nm were observed, with the size growing larger over the next few hours. The average diurnal variation in the aerosol number distribution in Fig. 12a shows that NPF events usually occur during the morning between 12:00 and 15:00 UTC (06:00–09:00 LT, local time), followed by particle growth during the rest of the morning and afternoon. This variation is also seen in the diurnally averaged CPC measurements of aerosol diameters >3 and >10 nm (Fig. 12c), but diurnal changes in CCN number concentrations (Fig. 12d) are more modest.

Various NPF pathways associated with different chemical species have been proposed and implemented in models. Two NPF pathways are considered in MAM4 in EAMv1: a binary nucleation pathway and a PBL cluster nucleation pathway. However, the current simulation does not reproduce the observed large day-to-day variability in small particle concentrations due to NPF. Instead, the model produces high aerosol concentrations between 10 and 100 nm almost all the time. It also fails to reproduce the large diurnal variability in the aerosol and CCN number concentration with a peak seen in the morning near 15:00 UTC (09:00 LT), 7 h earlier than the observed 22:00 UTC (16:00 LT) afternoon peak. Its overestimation of the aerosol number concentration for particle diameters >10 nm and its underestimation of the CCN number concentration is consistent with the information shown in Fig. 5. Several efforts are underway to improve the simulation of NPF by adding a nucleation mode in MAM4 to explicitly resolve ultrafine particles and by implementing new chemical pathways to simulate NPF following Zhao et al. (2020). ESMAC Diags is being used to evaluate these new model developments.

Using aircraft measurements from ACE-ENA, Zheng et al. (2021) recently found evidence of NPF events occurring in the upper part of the marine boundary layer between broken clouds following the passage of a cold front. The 16 February 2018 is identified as a typical NPF day in Zheng et al. (2021). The vertical profiles of aerosol number and CCN concentrations measured by aircraft on 16 February 2018 are shown in Fig. 13. The NPF event and particle growth that occurred in the upper boundary layer are shown by the large mean and variance in the aerosol number concentration just below the base of the marine boundary layer clouds. EAMv1 could not simulate NPF events in the upper marine boundary layer on this day and other days during ACE-ENA, likely due to the lack of NPF mechanisms related to effective removal of existing particles, cold air temperatures, vertical transport of dimethyl sulfide (DMS), and high actinic fluxes in broken marine boundary layer clouds (Zheng et al., 2021). Similarly, the sharp increase in the CCN number just above the level of marine boundary layer clouds is not simulated.

Unlike some field campaigns (i.e., HI-SCALE and ACE-ENA) in which aircraft missions were conducted over a relatively localized region with limited spatial variability in the meteorological conditions, ship and/or aircraft measurements over the NEP and SO test bed regions span regions >1500 km (i.e., from California to Hawaii and from Tasmania to the far Southern Ocean, respectively). As shown in Fig. 3, there are large spatial gradients in EAMv1-simulated aerosol optical depth along these ship/aircraft tracks. In ESMAC Diags version 1, we include composite plots of aerosol and cloud properties binned by latitude to assess model representation of synoptic-scale variations.

The research ship (aircraft) from the MAGIC (CSET) field campaign in the NEP test bed traveled between California and Hawaii, where there is frequently a transition between marine stratocumulus clouds near California and broken trade cumulus clouds near Hawaii (e.g., Teixeira et al., 2011). Although ESMAC Diags v1 focuses primarily on aerosols, we show some basic meteorological and cloud fields here, as they are important to illustrate the transition of cloud regimes along the ship (aircraft) tracks. Additional cloud properties derived from surface and satellite measurements are not included in the current analysis, but they are being implemented in ESMAC Diags v2. Some of the meteorological, cloud, and aerosol properties along the ship (aircraft) tracks binned by latitude are shown in Fig. 14 (Fig. 15). Note that the cloud fraction in Fig. 15 is calculated as the cloud frequency in the aircraft observations and from the grid-mean cloud fraction in the model along the flight track. This is different from the classic definition of cloud fraction usually used for satellite measurements or models and is subject to aircraft sampling strategy. As the surface temperature increases from California to Hawaii (Fig. 14a), the cloud fraction (Fig. 15a) shows a decreasing trend southwestward, indicating the transition from stratocumulus to cumulus clouds. However, the ship-measured liquid water path (LWP, Fig. 14b) has no trend related to latitude, possibly because cumulus clouds at lower latitudes have a smaller cloud fraction but a larger LWP when clouds exist. EAMv1 shows decreasing trends in both cloud fraction and LWP from high to low latitudes along these tracks. It generally underestimates the LWP and overestimates the cloud fraction to the north of 30∘ N. For aerosol number concentrations, EAMv1 produces too many aerosols compared with measurements, both at the surface (ship) and aloft (aircraft), consistent with the aerosol size distribution in Fig. 5 and the total number concentration in Table 3. However, EAMv1 does reproduce the increase trend in the accumulation-mode aerosol concentration approaching the California coast.

Similar latitudinal gradients of aerosol and CCN number concentrations along ship tracks from MARCUS and aircraft tracks from SOCRATES are shown in Figs. 16 and 17, respectively. Over the SO region, NPF frequently occurs during austral summer when ample biogenic precursor gases (e.g., DMS) are released and rise into the free troposphere (McFarquhar et al., 2021; McCoy et al., 2021). Large values of ship-measured aerosol and CCN number concentration are observed near Antarctica, corresponding to the coastal biological emissions of aerosol precursors, and also occur to the north of 45∘ S, indicating impacts from continental and anthropogenic sources. This is consistent with other studies (Sanchez et al., 2021; Humphries et al., 2021). EAMv1 underestimates the aerosol and CCN number concentration near Antarctica. This bias, which may be related to overly strong wet scavenging or insufficient NPF and growth, is commonly seen in many other ESMs (e.g., McCoy et al., 2020; McCoy et al., 2021). Aircraft flight paths during SOCRATES (Fig. 17) do not extend as far south as the ship measurements (Fig. 16). The observed aerosol properties generally have little latitudinal variation. EAMv1 underestimates the aerosol number concentration for particle sizes >10 nm and CCN number concentration with SS=0.5 %, but the predictions are closer to observed values for aerosol sizes >100 nm and CCN with SS=0.1 % (Fig. 17), consistent with the mean aerosol size distribution in Fig. 5. This indicates that the model performs better in simulating accumulation-mode particles than Aitken-mode particles over SO. These model aerosol biases are highly relevant when considering their interaction with clouds and radiations, which will be included in version 2 of ESMAC Diags.