The R component of the STILT model exists as a group of core functions used to track particle locations, calculate footprints, and apply surface flux grids. User groups have built upon these functions, adding scripts for common modeling workflows and additional functionality. Key components of the higher-level functions remain unpublished and undocumented prior to this paper, including a description of methods used to aggregate the particle ensemble to calculate footprints. Here, we adopt a widely used collaborative software development platform (GitHub) as a common source code repository that meets the needs of STILT users. This repository is built upon existing advection and dispersion calculations but has restructured and modernized the core functions used to interact with the model (Fig. 1).

A single script (run_stilt.r) defines model inputs such as receptor locations and meteorological fields, controls and executes the parallelized model, and outputs footprints. Footprints are saved in a netCDF format consistent with conventions for Climate and Forecast metadata (cfconventions.org), the standard for gridded model datasets by the University Corporation for Atmospheric Research (UCAR). This format is compatible with most popular data analysis software platforms and facilitates analysis of model output. The script run_stilt.r serves as the primary STILT interface, interacting with R functions, which in turn call Fortran subroutines for the bulk of calculations, and providing a systematic, well-documented workflow for users.

Executing simulations in parallel is essential to leverage the full capability of computing resources. STILT receptors are defined in a table of space (x,y,z) and time (t) coordinates enabling users to fix a receptor in space and model the time evolution of the influence field, distribute receptors across space and capture a snapshot at a single time, or distribute the receptors across both space and time. Since each STILT simulation is computationally independent, total simulation time can be reduced by distributing batches of simulations among parallel threads (Fig. 2). However, past methods for parallelizing simulations require users to manually define batches of receptors and relevant meteorological inputs in unique initialization scripts and submit each script as a separate job to the scheduler. While increasing the number of parallel threads decreases the size of each simulation batch, the requirements of the user become more complex.

We formalize methods for automatically distributing batches of receptors across many parallel threads managed by the model rather than the user. Within-node parallelism is achieved through process forking, in which batches of receptors are allocated across multiple parallel threads on a single machine. Multi-node parallelism is accomplished by interfacing with the Simple Linux Utility for Resource Management (SLURM), an open-source tool that provides the framework for interfacing with clusters of computer nodes . While SLURM is the only cluster job scheduler that has been implemented to date, the open-source code can be modified to run on systems managed by other job schedulers including TORQUE/OpenPBS, Sun Grid Engine, OpenLava, Load Sharing Facility, or Docker Swarm using methods described by . SLURM allocates computational resources with low overhead and can be used to dispatch job arrays of STILT simulations to multiple nodes. SLURM is used to parallelize among nodes and process forking by the modified STILT framework is used to parallelize within nodes. Process forking can be used independently to execute parallel simulations on a single machine or combined with SLURM to parallelize simulations within each SLURM node. Provided that memory limits are not exceeded, these methods enable total simulation time to decrease linearly with available CPU cores.

The influence of surface fluxes on air arriving at the receptor depends upon vertical dilution within the atmospheric column. The STILT model determines the height of the boundary layer zpbl using a modified Richardson number method . In the original STILT model, surface fluxes are instantaneously diluted within an effective mixing depth of h*=0.5⋅zpbl for which the vertical mixing timescale is comparable to the model time step for advection . As described by , an atmospheric column of height h(x,y,t,p) is used to relate surface fluxes F(x,y,t) to the mole fraction influence S(x,y,t,p) for each particle p as S(x,y,t,p)=F(x,y,t)mairh(x,y,t,p)ρ‾(x,y,t,p)z≤h0z>h, where ρ‾ is the average air density below h and mair is the molar mass of dry air. Thus, particles below h perceive surface fluxes diluted within an atmospheric column of depth h. However, the advective timescale is often too short for complete turbulent mixing of HNF fluxes to h before arrival at the receptor. While the assumption that surface fluxes can instantaneously mix to h* has been validated within the traditional near-field domain , this method underrepresents the influence of HNF fluxes on the tracer mole fraction arriving at the receptor.

We apply a method of calculating the effective mixing depth in the HNF based on homogeneous turbulence theory, described by . Each model time step component k of the HNF mixing depth h′ at time tk will be of the form hk′(p,t)=zr+∫0tkσw2TLt+TLe-tTL-1dt, where zr is the height above ground of the receptor, σw is the standard deviation in vertical velocities encountered by p during the integration time step, and TL is the Lagrangian decorrelation timescale. Substitution of h=min⁡(h′,h*) into Eq. (1) enhances the mole fraction influence of HNF sources on the receptor (Fig. 3). The time step at which h′≈h* signifies the transition between the HNF and traditional near-field domains. The formulation of h′ grows the particle-specific dilution depth relative to local turbulence and enables the extent of the HNF domain to vary depending on receptor location, meteorological conditions, and local topography. For surface-based applications, h′ grows to h* over 0.1–1 h, affecting a spatial domain of 1–10 km adjacent to the receptor.

Computing trajectories of large particle ensembles (N>104) is computationally expensive. To lessen the cost of each simulation, footprint fields are often calculated from smaller particle ensembles by applying smoothing methods to compensate for the smaller ensemble size. These smoothing methods are less computationally expensive than calculating trajectories for a larger ensemble but vary in their ability to reproduce the robust footprint field of the large particle ensemble.

Prior to methods described in this section, STILT footprints have been calculated by accumulating the influence of particles over an averaging volume. To lessen grid noise from the few particles spread throughout the grid, the spatial extent of the particle ensemble was used to dynamically coarsen the size of the averaging volume by a factor of 2 as the particle cloud spreads, first shown in . However, at finer resolutions, this method results in excessive smoothing, removing information calculated by the advection and dispersion routines (Fig. 4).

Here we introduce a kernel density estimator to spatially allocate the influence of particles to the footprint grid and show improvements over the prior method at fine spatial resolutions. This method distributes the influence of each particle using a Gaussian weighted spatial kernel centered over the particle's position. The size and intensity of the spatial kernels are defined by the kernel bandwidth, which is determined at each model time step using elapsed time and total dispersion of the stochastic ensemble as proxies for uncertainty in the locations of individual particles. As model time or total ensemble dispersion increase, the kernel bandwidths increase the amount of smoothing applied to each particle. Dispersion of the particle cloud at each time step is represented using a nondimensionalized standard deviation of particle locations σd(t) given by σd(t)=σx2(t)+σy2(t), where σx2(t) and σy2(t) are Euclidean variances in the x and y positions of all particles in the ensemble at time t. We find σd(t) to agree with other metrics of dispersion within the particle ensemble, such as the average pairwise distance (r2>0.99), with less computational expense. The bandwidths of the Gaussian smoothing kernels are then given by b(t)=f0.06t⋅σd(t)cos⁡(ϕ(t)‾), where t is time elapsed in days, ϕ(t)‾ is mean latitude of the particle ensemble used for approximation for meridional grid convergence, and 0.06 is an empirically derived constant. f defaults to 1 and is provided as a user-defined smoothing adjustment to enable manual manipulation of kernel sizing.

We test the new footprint calculation methods against a brute force simulation with an atypically large particle ensemble size (N=105) aggregated over a homogeneous grid (Fig. 4). This large simulation is computationally expensive but generates an idealized, physically constrained footprint without smoothing algorithms. The simulation receptor was positioned at a SLC CO2 measurement site on a summertime afternoon and particles were followed backward in time for 24 h. We then demonstrate differences between the new kernel density estimator and the traditional dynamic grid-coarsening footprint calculation methods (Fig. 3) for a typical particle ensemble (N=200) and for an extreme case with atypically few particles (N=10). The effects of varying the smoothing parameter (f=1,2) are shown and errors are quantified using the difference of calculated grid cells from the idealized brute force case.

For the typical case (N=200 and f=1), the kernel density estimator shows improved agreement with the brute force method (rmse=5.60×10-4 ppmµmol-1m2s) compared to the traditional dynamic grid-coarsening method (rmse=5.79×10-4 ppmµmol-1m2s), preserving a Gaussian plume adjacent to the receptor, a clustered area of high influence, and capturing split flow upstream. When the kernel bandwidths are doubled by increasing the smoothing parameter (f=2), the footprint field becomes over-smoothed and becomes less similar to the brute force case (rmse=5.66×10-4 ppmµmol-1m2s). In the extreme case using atypically few particles (N=10), the dynamic grid-coarsening method produces a footprint field dominated by noise from individual particles (rmse=6.12×10-4 ppmµmol-1m2s). The kernel density estimator (f=1) improves results but shows fragmentation further from the receptor (rmse=5.75×10-4 ppmµmol-1m2s). In this case, the kernel density estimator smoothing parameter enables users to manually widen the plume reproduced in the footprint. Doubling the smoothing parameter (f=2) improves similarity to the smaller particle ensemble (rmse=5.70×10-4 ppmµmol-1m2s) and demonstrates how users can modify the kernel bandwidths to adapt the model to unique cases. While tracer mole fraction differences between the two footprint calculation methods vary depending upon the locations of footprint differences relative to sources, tracer mole fractions calculated using the kernel density estimator are more similar to the idealized brute force case.

We demonstrate these changes to STILT by comparing CO2 mixing ratios simulated by the STILT model with corresponding measurements onboard an electric SLC light-rail commuter train during July 2015. The Salt Lake Valley (SLV) is a 1300 km2 area encompassing SLC and its surrounding suburbs, bounded by the Wasatch Range to the east, the Oquirrh Mountains to the west, the Traverse Mountains to the south, and the Great Salt Lake to the northwest. A light-rail train is equipped to measure high-frequency (1 Hz) CO2 mole fractions in repeated transects of the SLV using a Los Gatos Research Ultraportable Greenhouse Gas Analyzer. CO2 and CH4 mole fractions are corrected for water vapor dilution and spectrum broadening and are calibrated every hour using a compressed whole air tank with known tracer mole fractions traceable to World Meteorological Organization standards. The light-rail train typically operates between the hours of 05:00 and 23:00 local daylight time (LDT) and only these hours were used in analyses. For details related to the measurement platform, refer to .

The observations generally show higher CO2 mole fractions in SLC's urban center and along the north–south-oriented urbanized corridor centered in the SLV (Fig. 7), consistent with urban spatial CO2 gradients observed in previous studies . The lowest mole fractions were observed in the southwest corner of the SLV at the margin of recent suburban developments. At a finer scale, the high-frequency measurements show mole fraction enhancements near busy roads and intersections. Measured mole fractions are also consistently higher along a 3 km section of the light-rail track running along the center of a busy six-lane road.

The Hestia bottom-up anthropogenic CO2 emissions inventory characterizes carbon fluxes by estimating emissions at the scale of individual buildings and roadways . Hestia is available for a handful of US cities including Indianapolis, Los Angeles, Baltimore–DC, and SLC. Details pertaining to the Salt Lake County Hestia product are described by . For this simulation, Hestia anthropogenic CO2 fluxes are aggregated hourly to a 0.002∘ grid (Fig. 6). However, the Hestia inventory only encompasses the SLV and requires the use of a larger-scale anthropogenic emissions inventory to account for fluxes originating from outside of the SLV.

We apply the 1 km × 1 km ODIAC inventory to account for anthropogenic CO2 emissions originating outside of the SLV. ODIAC is a globally available gridded dataset that uses power plant profiles and satellite-observed nightlights to spatially allocate estimates of total anthropogenic CO2 emissions. The gridded ODIAC data are temporally allocated to hourly time steps using methods described by .

Within the SLV domain where the inventories overlap, Hestia and ODIAC agree on the total anthropogenic emissions to within 1.5 % during our study period. However, uncertainties of fluxes applied to our analyses are likely larger since the two inventories allocate fluxes differently in space and time. Further, only Hestia is used to represent the SLV whereas ODIAC is used outside of the SLV to account for regional-scale emissions. Uncertainties in inventory estimates are difficult to quantify in time and space and require a devoted effort within the emission inventory scientific community to propagate uncertainties through underlying assumptions within each inventory .

The biological CO2 inventory determines land surface types using the 2011 National Land Cover Database and 1 m lidar-derived discrete land cover classifications across the SLV . The link between land cover classification and CO2 exchange is established using AmeriFlux eddy covariance data that provide a robust estimate of biologic fluxes from different vegetation types . A lookup table with independent axes for temperature, incoming shortwave radiation, and week of year is used to describe the relationship between land cover classification and biological fluxes over a 0.01∘ grid. For details pertaining to the biological flux inventory, refer to .

STILT receptors are defined by averaging light-rail measurements hourly over a 0.002∘ grid (roughly 200 m at ϕ=41∘ N). The grid resolution was chosen to pair analyses with the 0.002∘ Hestia inventory and because 0.002∘ corresponds roughly with the size of a SLC block. This method results in 33 608 unique receptors for the month of July 2015, necessitating the use of parallel simulations and fine-scale footprint calculation included in the modified framework.

Urban development and expansion in the area surrounding SLC is limited by the mountainous topography surrounding the city and the Great Salt Lake, which restrict the expansion of the city and suburbs. This confines large anthropogenic and biologic sources into a relatively small area surrounding the SLV and simplifies boundary conditions for SLV-centric modeling efforts. From each receptor, 24 h backward trajectories of 200 particle ensembles were calculated using meteorological fields from the HRRR model, available at an hourly interval with a 3 km grid resolution. On average, particles travel within the model domain for 11 h. Computation of the 33 608 particle trajectories and a single set of footprints was completed in 5.5 h. This calculation utilized 80 parallel threads across five nodes, each equipped with 64 GB of memory with two eight-core Intel Xeon E5-2670 2.6 GHz processors. Of the simulations, 6.7 % were not completed due to short-term outages in the HRRR data product.

We compute footprint fields using the legacy dynamic grid-coarsening (LEG) algorithm as well as Gaussian kernel density estimation with the HNF vertical mixing depth correction (GWD) and without the HNF vertical mixing depth correction (GND) to illustrate the differences among methods. Further, these three methodologies are applied for two model domains, resulting in six different permutations of footprint fields for each receptor. A fine-scale 0.002∘ grid encompasses the SLV and is used to apply SLV anthropogenic emissions. A 0.01∘ grid covering a larger area of northern Utah is used to apply biological fluxes and non-SLV anthropogenic emissions.

Footprints are convolved with anthropogenic and biological CO2 fluxes and added to background CO2 mole fractions that are representative of CO2 mole fractions that have not been influenced by urban emissions . The background mole fractions are taken from a nearby high-elevation measurement site at Hidden Peak at the top of the Snowbird ski resort in the Wasatch Range . We use a similar approach to prior studies (e.g., ) and focus this analysis on the afternoon and early evening hours (12:00–19:00 LDT) to lessen the influence of boundary layer development, nocturnal stratification of the boundary layer, and shallow turbulence on measured mole fractions that would not be represented in the 3 km resolution of the HRRR meteorological fields.

Observed and simulated mole fractions are averaged by hour of day to generate mean diel cycles, shown in Fig. 5. Observations show elevated mole fractions at night and early morning, decreasing into the afternoon as convective mixing increases . All three of the simulated diel cycles derived from the different footprint algorithms systematically underestimate nighttime and early morning mole fractions, consistent with previous studies . However, during afternoon hours the simulated values track more closely with the observations, with GWD exhibiting closer correspondence than GND and LEG (Fig. 5).

Explained variance over space between the time-averaged modeled and measured concentrations is highest for GWD (Pearson's r2=0.27) followed by GND (r2=0.21) and lastly LEG (r2=0.20). While we have demonstrated the Gaussian kernel methods to compare favorably with idealized footprints calculated with brute force particle simulations, this analysis found that modeled concentration differences between GND and LEG fall within the uncertainties in surface flux inventories. As GWD agrees most closely with observations over time and space, we focus on GWD for the remainder of analyses.

Footprints convolved with surface flux inventories (Fig. 6) show measurements made on the light-rail train to be highly sensitive to fluxes in the HNF domain. Spatially averaged model results capture the mole fraction gradient between the urban center and surrounding suburbs (Fig. 7). The lowest modeled mole fractions occurred in the southwest corner of the SLV, in agreement with measurements. The model generally produced mole fraction enhancements (ΔCO2) for grid cells containing or downwind from major roadways (Fig. 7). However, modeling intersection scale enhancements would require finer grid spacing capable of resolving sub-city-block spatial scales that is not yet feasible given current constraints on inventories, meteorological data, and computing resources. On average, we found the largest contributor to modeled CO2 mole fractions is the SLV anthropogenic fluxes (ΔCO2=4.18 ppm), followed by biological fluxes (ΔCO2=-0.89 ppm), and the smallest contribution is from non-SLV anthropogenic fluxes (ΔCO2=0.37 ppm).

Key differences between modeled and measured mole fractions exist near HNF sources at the sub-grid scale (Fig. 7). While the model does capture localized mole fraction enhancements near busy roads and intersections (“I” in Fig. 7), measured mole fractions are systematically higher than corresponding model estimates in these areas. These results indicate that the light-rail measurement platform is sampling emissions prior to mixing with the surrounding air. This is evident along the section of light-rail track that shares the six-lane road with other vehicles (“R” in Fig. 7) on which large discrepancies between measurements and model estimates are regularly observed. By diluting emissions throughout a larger grid cell, the model predicts elevated mole fractions localized within and downwind from cells containing significant sources but does not fully capture the magnitude of enhancement resulting from the close proximity to the emissions source.

To demonstrate the benefits of the 0.002∘ grid resolution, we use the above 0.002∘ grid as well as spatially degraded 0.01∘ and 0.1∘ grid resolutions to convolve footprints and fluxes (Fig. 8) and compare with light-rail measurements. Correlations over space between the time-averaged modeled and measured concentrations are highest for the 0.002∘ grid (r=0.52). We find that degrading the resolution to 0.01∘ still captures the SLV-scale urban–suburban–rural mole fraction gradient but fails to resolve much of the roadway and intersection scale enhancements, resulting in a modest decrease in agreement with measurements (r=0.48). However, degrading the resolution to 0.1∘ prevents the model from resolving much of the spatial mole fraction variation (r=0.25). Further, evaluating the spatially averaged concentration by hour of day shows improved agreement with measurements among the finer grid resolutions (0.002, 0.01∘) over the more coarse 0.1∘ resolution (Fig. 9). While all three resolutions mimic the temporal pattern in the observed mole fraction enhancements due to the temporal variability assigned to emissions inventories, the finer resolutions better capture the mole fraction enhancements observed by the light-rail train in both time and space.