The atmospheric Large Eddy Simulation (LES) formulation is based on the moist, compressible, filtered Navier-Stokes equations of gas dynamics with non-hydrostatic buoyancy-driven motions. The gas is treated as a sum of ideal gases, dry air and water vapor – with immersed hydrometeors that contribute to mass but not to pressure. The sub-grid-scale dissipation is assumed to be dominated by eddies, and this model uses a common eddy viscosity formulation . The equations are cast in Cartesian geometry as follows: 1∂∂tρρuρvρwρθρqvρKρqℓρqP+∂∂xρuρuu+p′+τ11ρuv+τ21ρuw+τ31ρuθ+τθ1ρuqv+τv1ρuK+τK1ρuqℓ+τℓ1ρuqP+τP1+∂∂yρvρvu+τ12ρvv+p′+τ22ρvw+τ32ρvθ+τθ2ρvqv+τv2ρvK+τK2ρvqℓ+τℓ2ρvqP+τP2+∂∂zρwρwu+τ13ρwv+τ23ρww+p′+τ33ρwθ+τθ3ρwqv+τv3ρwK+τK3ρwqℓ+τℓ3ρwqP+τP3=000-ρ′g00KS+KD+KB02p=ρdRd+ρvRvT=ρR⋆T=C0ρθγ;θT=R⋆Rdp0pRd/cp;R⋆=ρdρRd+ρvρRv where ρ is total density; u, v, and w are wind velocities in the x, y, and z directions, respectively; θ is a form of virtual potential temperature defined by Eq. (); qv is the wet water vapor mixing ratio (such that ρv=ρqv is the density of water vapor); K is unresolved, sub-grid-scale specific Turbulence Kinetic Energy (TKE); qℓ is a tracer quantity that contributes to mass but not to pressure; qP is a passive tracer that is mass-weighted but contributes to neither mass nor pressure (e.g., mass-weighted number concentration for two-moment microphysics); p is total ideal gas pressure defined by Eq. (), a sum of dry and water vapor pressures; C0=Rdp0-Rd/cpγ is a constant of proportionality; γ=cp/cv is the ratio of specific heats of dry air; cp is specific heat of dry air at constant pressure; cv is specific heat of dry air at constant volume; p0 is reference pressure; Rd is the dry air ideal gas constant; Rv is the water vapor ideal gas constant; KS is TKE shear production; KD is TKE dissipation; KB is the TKE buoyancy source/sink; τij∀i,j∈1,2,3 is the unresolved eddy flux of wind momenta; τθj∀j∈1,2,3 is eddy flux of potential temperature; τvj is eddy flux of water vapor; τKj is eddy flux of TKE (i.e., the “turbulence transport” terms for TKE); τℓj is the eddy flux of tracers that contribute to mass but not to pressure (e.g., hydrometeors); τPj is the eddy flux of passive mass-weighted tracers contributing to neither mass nor pressure; and g is acceleration due to gravity. The total density, ρ, is defined as the sum of all mass-contributing densities: ρ=ρd+ρv+∑ℓρℓ, where ρd is the density of dry air. The quantities in Eq. () are filtered quantities.

The perturbation pressure, p′=p-pH, and perturbation density, ρ′=ρ-ρH, are deviations from a dominant hydrostatic balance denoted by: 3dpHdz=-ρHg which can be defined arbitrarily. In these simulations, the balance is obtained with either a vertical potential temperature profile or a combination of a vertical temperature profile and a vertical water vapor dry mixing ratio profile. The purpose of removing hydrostasis is to better resolve the perturbations, which form the primary means of forcing for buoyancy-driven atmospheric flow. Poor resolution of hydrostasis results in spurious vertical velocities, which can render the solution physically unrealizable.

Equation set () will also be expressed in places using a generic vector form of conservation laws: 4∂q∂t+∂f∂x+∂g∂y+∂h∂z=s

Using the vector form of the LES equations in Eq. (), we split the equations such that each dimension is handled separately, and the vector form equations become, in general: 14∂q∂t+∂f∂x=s This is integrated in space (semi-discretized) over domain-spanning cells of equal spacing, Ωi=xi-1/2,xi+1/2, where xi±1/2=xi±Δx/2. The divergence theorem is applied, and an upwind Riemann solver is used to give the form: 15∂q‾i∂t+Rq̃ixi+1/2,q̃i+1xi+1/2-Rq̃i-1xi-1/2,q̃ixi-1/2Δx=sq‾i where q̃ix is a reconstruction of the variation of q within the cell of domain Ωi, Rq-,q+ is a Riemann solver that returns a flux vector, fq, from multi-valued states at cell edges using the upwind state based on locally frozen characteristics (the flux Jacobian diagonalization: ∂f/∂q=L-1ΛL). The state q-+q+/2 is used to compute the diagonalization for upwinding. For simplicity, a constant acoustic speed of 350 m s⁻¹ is assumed, and supersonic flow is not supported in the Riemann solver. This is the same approach used in .

While the gravity and eddy viscosity source term can be integrated with high-order discretizations, the hyperbolic dynamics use constant and uniform gravity, and the diffusive SGS dynamics do not benefit substantially from high-order accuracy. For the reconstruction, q̃ix, eleventh-order Weighted Essentially Non-Oscillatory (WENO) limiting is used along with weight mapping . When a cell is within ten cells of an immersed boundary in any direction, weight mapping is omitted to reduce oscillations because immersed boundaries essentially represent permanent contact discontinuities.

The optimal, three-stage, third-order accurate Strong Stability Preserving Runge-Kutta (SSPRK3) time discretization is used to integrate the remaining temporal Ordinary Differential Equation resulting from the semi-discretization in Sect. . The authors experimented with different dimensional splitting approaches, and it was found that the least numerically diffused results came from splitting the dimsensions fully independently within each stage of the SSPRK3 time stepping. While this limited the maximum stable CFL value to roughly 0.7, it resulted in less numerical dissipation than using dimensional splitting techniques outside the SSPRK3 time stepping procedure. This approach with SSPRK3 also produced less numerical diffusion than attempts at using an ADER time discretization with Differential Transforms with dimensional splitting outside the ADER computations. This approach to dimensional splitting within each RK stage has also been shown to reduce errors at singularities on non-orthogonal grids such as the cubed-sphere .

The vertical grid in portUrb can be arbitrarily set by providing a set of monotonically increasing vertical interface levels. This is supported simply in the numerics by use of a metric transformation from the physical vertical coordinate, z, to an equally spaced reference coordinate: ζz. For convenience, the ζ coordinate is simply set to be the vertical interface index such that the grid spacing in ζ coordinates is always one. Before reconstruction, the cell average, qk‾, is multiplied by the physical grid spacing, Δzk, to transform into ζ coordinates (recalling that Δζ≡1). Then, after reconstruction in ζ coordinates, the cell-edge values are multiplied by the inverse of a high-order reconstruction of dz/dζ at the cell edge in question to transform back into physical vertical coordinates.

The portUrb code has several convenience functions to generate a stretched vertical grid. There is a function to generate equal grid spacing. There is a function that generates exponentially increasing vertical grid spacing based on an input of the vertical domain extent and the desired number of vertical levels. The routine fits the constant multiple by which grid spacing increases between successive levels to those parameters. There is also a function that takes an input of a lower level grid spacing and a higher level grid spacing along with the vertical domain extent and the domain over which the grid spacing transitions between the two using smooth polynomial interpolation. This routine determines the number of vertical grid cells rather than having that value specified by the user.

There are four different boundary conditions that can be specified in the dynamical core and SGS TKE closure routines: periodic, solid wall, open, and precursor forcing. The precursor boundary forcing is implemented by declaring one coupler object to be the concurrent precursor simulation. That simulation then saves the state in the domain's “ghost cells” for use by the forced simulation for each of the Runge-Kutta stages. Then, the saved ghost cells from the precursor simulation are copied into the forced simulation's ghost cells at each Runge-Kutta stage if the velocity normal to the boundary is flowing into the domain. An open boundary is used if the velocity normal to the boundary is flowing out of the domain.

The model is constructed with a “coupler” at the core. The coupled state is defined to be dry air density, wind velocities in all three directions, absolute temperature, and tracer densities. For each tracer, it is defined whether the tracer should remain positive, whether it contributes to mass, and whether it is diffused with the SGS closure. The only required tracers are water vapor density (for all simulations) and mass-weighted TKE (if the LES closure module is used). All tracers must be specified as mass-weighted or total density. This coupled state has the benefit of being non-ambiguous (whereas definitions of potential temperature and mixing ratios can be varied). A disadvantage computationally is that most parameterizations and dynamical cores use mixing ratios and potential temperature, meaning a conversion is needed before and after every module, typically. This cost is deemed acceptable to have a non-ambiguous coupler state.

A “module” is a self-contained action that changes the coupled state or produces output, and all model actions are implemented as modules. The coupler state is passed to a module along with the desired time step. Modules may sub-step to maintain stability as needed, but in the experiments in this study, the dynamical core imposes the smallest time step, and all modules are run at the dynamical core time step for now. The initial state, therefore, is implemented based on the coupler's state rather than the dynamical core state defined in Eq. ().

Ensembles within a single executable are flexibly supported with a core module called an “Ensembler”, a class that manages ensembles. One can implement different “dimensions” of ensembles, specify how many tasks (relative to the smallest ensemble member) a given member needs, split the overall MPI communicator into sub-communicators (one for each ensemble member), and redirect the output of each member to its own file. This way, each ensemble member can do any arbitrary operation, read from any individual file, and output to any arbitrary file. With this approach, one can submit capability-class jobs (that is, jobs that use 20 % or more of the machine at a time) on supercomputers such as DOE Leadership Computing Facilities like the Oak Ridge Leadership Computing Facility or the Argonne Leadership Computing Facility. This capability is used in several of the numerical experiments in Sect. .

The model is coded in a portable C++ library called Yet Another Kernel Launcher (YAKL) , which is based on the Kokkos portable C++ library . The core of these libraries is to wrap kernel code (ostensibly the code inside a set of loops over the grid cells being operated on) in a class object. That object is then passed to a launcher that launches the code in parallel on a given architecture. For Nvidia, AMD, and Intel GPUs, the CUDA, HIP, and SYCL specifications are used to launch the code, respectively. There are other backends available as well such as OpenMP threading for multi-core CPUs. All modules in portUrb that operate on a set of grid cells are run on whatever “device” is available on a given machine (typically a GPU). Care has been taken to ensure all kernels run efficiently by streaming memory effectively from DRAM (ensuring higher GPU memory bandwidth is realized) and keeping kernel resource usage low enough to use the entire GPU effectively (a concept known as “occupancy”) .

This test case compares LES solutions of flow past a large array of surface-mounted cubes against results from a wind tunnel experiment . The simulation approximates conditions at the center of the cube array, where measurements were made, by using periodic horizontal boundary conditions with four staggered cubes. The cubes are of a length, width, and height of h=20 mm. The domain is 4 h × 4 h × 10 h in the x, y, and z directions, respectively. The bottom and top boundaries are free-slip solid walls with isentropic thermal conditions. A kinematic viscosity of ν=1.5×10-5 m² s⁻¹ is used, and a grid spacing of 1 mm is used in all directions. Output is performed every 0.1 s, and the simulation is run for 5 s total.

Gravity is not enabled for this simulation, initial pressure is 105 Pa, initial temperature/potential temperature (they are the same in this case) is 300 K, and initial velocity is 10 m s⁻¹ in the x direction and zero in the y and z directions. A pressure gradient forcing is used to penalize deviations from 5.82 m s⁻¹ in the x direction at a height of 0.043 m (the average of the observed velocities at the highest points above the ground) over a time scale of 100 time steps. A surface flux is applied with a roughness length of 10-7 m at the model surface and 10-6 m at cube surfaces. Surface roughness lengths in the set 10-7,10-6,10-5 were tested, fully tensored in 9 experiments, for the model surface and cube surfaces using portUrb's ensemble capability.

The four cubes are placed on the following domains: (1) h/2,3h/2×h/2,3h/2×0,h; (2) h/2,3h/2×5h/2,7h/2×0,h; (3) 5h/2,7h/2×3h/2,5h/2×0,h; and (4) 5h/2,7h/2×0,h/2×0,h∪5h/2,7h/2×7h/2,4h×0,h. In the experiment, the flow is sampled at various vertical columns at four horizontal locations, each of which have four corresponding points of geometric similarity in the domain. The four point locations are: (1) P0=h,3h, which corresponds to the center of a cube location; (2) P1=2h,3h; (3) P2=h,2h; and (4) P3=2h,2h.

Given there are four cubes, each of these points have three other points that are geometrically identical inside the domain (due to periodic horizontal boundary conditions and symmetry of cube placement within the domain), creating four points total at which P0, P1, P2, and P3 will be each be sampled – the results of which are averaged in order to compare against wind tunnel observations. These other three geometrically identical points correspond to shifts of -2h,0; 2h,h; and 2h,-h – where periodic wrapping is used to remain in the domain. Where points lie on a domain boundary, the smallest location magnitude (zero) is used.

The time-averaged u-velocity and u′w′‾ outputs over intervals of 0.1s are all interpolated to observation locations and compared against observations to determine the best fits. The one second period with the best fit according to a relative L2 norm averaged among the six comparisons (four u‾ comparisons and two u′w′‾ comparisons) is used to compared against observations. The time period 0.4s,1.4s is used for comparison.

Figure shows the placement of cube isosurfaces as well as Q-criterion to visualize the vortices in the flow at t=1 s. Figure plots these quantities against observations at each of the points. The perturbation quantities u′ and w′ are temporal deviations from the current simulation-spanning temporal mean and are accrued at each grid cell and each time step. The time-averaged u velocity and u′w′‾ correlations are both well captured by the LES simulation during the performance time period. The u′w′‾ extrema above the cube surface at P1 is not well-captured in terms of scale (an error common to other LES simulations of this case as well), but the structure and existence of a local maximum is captured. The correlation at P3 is very well captured up to 1.5 cube heights (particularly the slope up to the maximum and the maximum magnitude as well), but it is underestimated above that height. In summary, comparison against anemometry data shows that the LES model, the immersed boundary implementation, and the surface friction implementation are well formulated for capturing realistic turbulent dynamics around obstacles.

In order to tune the immersed boundary approach for partially immersed cells, a sphere is placed in a domain with periodic conditions in the y and z directions with specified laminar inflow/outflow boundaries in the x direction and the gravity term (and hence thermal and mass stratification) turned off. The domain is 2 km × 1 km × 1 km in the x, y, and z directions, respectively. The sphere center is located at 0.4 km in the x direction and is centered in the y and z directions with a radius of 100 m. A wind velocity of u=10 m s⁻¹, v=w=0 is specified at x-direction boundaries through a horizontal sponge layer over 2 % of the domain length with a decay of x3 to serve as inflow and outflow non-reflecting boundaries.

A high resolution reference solution is performed at 10 m grid spacing (20 cells across the sphere diameter), and low resolution runs are performed at 50 m grid spacing (4 cells across the sphere diameter) to test different exponents to the immersed proportion in Eq. (). Testing integer increments of the exponent in Eq. (), the value of five was demonstrated to give the lowest errors relative to the high resolution time-averaged solution coarsened to the low resolution grid. The time average is taken between 15 and 60 min of a 60 min simulation. The sum of differences for each individual time-averaged velocity component is used for the comparison between high and resolution solutions.

A plot of stream tubes flowing past the sphere at high and low resolution is given in Fig. . The low resolution solution has lower velocities because the cells near the sphere are coarser and have partially immersed material in them, slowing down the flow near the sphere. The flow slightly further away from the sphere is also averaging in lower wind speeds in the coarser-scale cells.

To observe the character of the flow through realistic buildings and the influences of building friction, this test case performs flow through a subset of buildings in Manhattan southwest of Central Park, rotated by 29° (to achieve a roughly North-South orientation of the buildings) over a domain of 1.31 × 1.716 × 0.572 km in the x, y, and z dimensions. There are two primary goals of these simulations: (1) to understand if the code is well-posed for this kind of flow, free from significant numerical artifacts, and giving physically realizable results; and (2) to demonstrate the influences of different building roughness lengths on flow through and above the buildings. Eleveth-order WENO interpolation is used for this simulations with no modifications from the other simulations in this study.

To ingest building geometry, the Blosm (https://github.com/vvoovv/blosm, last access: 1 December 2025) plugin to the application Blender (https://www.blender.org/, last access: 1 December 2025) is used to bring in 3-D building geometry from OpenStreetMap (https://www.openstreetmap.org/, last access: 1 December 2025; https://openstreetmap.org/copyright, last access: 1 December 2025) (the data from which is available under the Open Database License or ODbL), which automatically transforms WGS84 latitude-longitude coordinates into local East-North-Up (ENU) Cartesian coordinates in units of meters. Buildings outside the rectangular horizontal area of interest are removed, and the file is saved as a triangulated Wavefront “obj” file (meaning only triangles are allowed as surfaces rather than generic polygons). The triangulated mesh is then read into portable C++ using custom code, ignoring material properties and vertex normals – essentially only storing a set of triangular faces, each defined by three vertices in 3-D space.

Finally, for each horizontal point over a 9 × 9 point Gauss-Legendre-Lobatto (GLL) grid of points within each horizontal cell, the maximum height of any intersecting mesh triangle at a given point is computed and stored. During initialization, anything below that maximum point is declared to be embedded, and the proportion of embedded material in each cell is computed with GLL quadrature in all dimensions. While this process could potentially be improved by doing raycasting to determine non-monotonically embedded regions in the vertical, this requires a well-posed mesh to avoid incorrect detection of which portions of the vertical are immersed and which portions are not. Because of the extra mesh requirements and reliance on external tools to create these meshes, this is left for future research. The extension is straightforward, however, as raycasting is a relatively cheap and simple process.

A 20 m padding is added to the horizontal direction – roughly the mean distance between blocks of buildings in this domain, and a 100 m padding is added above the highest building in the domain, which is 470 m. A neutrally buoyant atmosphere is specified throughout the vertical domain with a sponge layer applied to the top 2 % of the domain to absorb waves that would otherwise reflect. A wind speed of 20 m s⁻¹ from the West (rotated by 29° to account for rotated building geometry) is initialized everywhere except in immersed cells. Deviations from a horizontal mean of 20 m s⁻¹ flow from the West at a height of 500 m are penalized to enforce a consistent wind speed throughout the course of the simulation (i.e., pressure gradient forcing). Periodic boundary conditions are applied in the horizontal direction to quickly spin up turbulence, essentially assuming infinite repetition of the city block geometrically. In the vertical direction, free-slip boundaries are applied with a surface friction at the bottom boundary specified by a roughness length of 0.05 m.

Three different building material roughness lengths are used (all sharing the same surface roughness length): 10-6, 0.05, and 0.5 m. The value of 10-6 m is essentially free-slip when using a grid spacing of 2 m; 0.05 m is likely the roughly physical value (accounting for unresolved building features like window insets, bricks, etc.); and 0.5 m is likely unrealistically large. Simulations are performed for 100 min: a 10 min spin-up period followed by a 90 min simulation period for time-averaging and comparison between different building roughness lengths. On 120 Nvidia A100 GPUs, the 100 min simulations took about 24 h of walltime a piece to complete, with about 1.3 million cells per GPU (161 million cells total). 2.25 million time steps were required to simulate 100 min in model time.

Figure gives iso-surface plots of the immersed proportion (i.e., buildings) in white, and Q-criterion in blue (shaded by velocity magnitude) at t=100 min of simulation using a building surface roughness of 0.05 m. Because Q criterion is plotted at a positive iso-surface value, vortex-dominated flow is shown rather than shear-dominated flow – thus the gap between buildings and visualized vortices as sheared flow transitions to turbulent vortices on the lee side of buildings.

Figure plots, at z=1 m, the 90 min time-averaged velocity magnitude at a building surface roughness of 10-6 m, (uz0=10-6), where discontinuities are the most likely to show up in the velocity time average. It also plots the difference, uz0=10-6-uz0=0.05, as well as the difference, uz0=10-6-uz0=0.5, to show how much slower the flow through frictioned buildings is. This is also plotted for z=101 m in Fig. . In these plots, there is no sense of any discontinuities present in the solution at either of the vertical levels. As the building friction increases, as expected, the flow becomes slower through the buildings, though not universally so (even in the 90 min time average) due to the turbulent nature of the flow. Also, as expected, the decrease in flow velocity is concentrated more highly at building edges and is more diffuse farther away from buildings.

Street canyon ventilation is generally greater with less frictioned buildings as expected, meaning pollutants and temperature extrema are likely more effectively mixed out of the surface layer. This is evidenced by the velocity magnitudes being larger with lower friction but also by the increased resolved and unresolved TKE shown in Figs. and , respectively. These plots indicate that fluctuations are also generally greater with lower building surface friction. There is a location at around x≈260 m, y∈500,600 m, where the surface experiences significant mean velocity and fluctuations in the 90 min average, showing an interesting convergence from induced vertical flow through the buildings. It is not the only location of this nature, but it is the most significant, showing that mean surface-level flow can be quite intermittent spatially in urban environments both in mean values and in fluctuations.

While a realistic flow through urban settings would clearly require tuning the building and surface friction terms to match observed flow, this test case demonstrates physically realizable results coupled with expected outcomes regarding mean flow rate and fluctuations in an urban setting when perturbing the building surface friction settings with realistic geometry. Also, it appears that complex building geometry can be ingested directly into the model without any need for smoothing, pre-processing, or conditioning. The use of WENO limiting, an upwind Riemann solver, and slight softening of building edge immersed cells in terms of forcing time scale appear to be enough to handle any generic building geometry while avoiding visible artifacts in the solution.

This test case simulates the same building setup as Sect. except with a larger domain and the use of a concurrent turbulent precursor for forcing the inflow boundaries, open boundaries for outflow, and a stretched vertical grid reaching up to 1800 m. The city setup covers a domain of 1269 × 1675 m × 472 m in the x, y, and z dimensions, respectively. To allow for a well-posed turbulent precursor forcing, the domain size is doubled in the horizontal dimensions with the city placed in the center; and the vertical domain is extended to 1800 m in order to simulate the interaction of the city with the ABL inversion layer.

The horizontal grid spacing is set to 4 m. The vertical grid is stretched from approximately 4 m grid spacing in the domain z∈0,480 m to a grid spacing of 10 m in the domain z∈600,1800 m, where z∈480,600 m is the transition region. The transition region is over approximately 18 cells, and the grid spacing transition region is plotted in Fig. . The same initial conditions in the neutral ABL test case in Sect. is used for this simulation as well including the geostrophic forcing to create an Ekman spiral. The simulation uses 137 million cells simulated over ten model hours to fully develop the ABL.

Figure plots the instantaneous horizontal wind speed at t=10 h for the concurrent turbulent precursor simulation (left) and the forced simulation with the city included (right) at z=100 m (top) and at y=1676 m (bottom). From these plots, it is clear that the precursor forcing is working correctly, as the same wind patterns are observed in the inflow at the left of the top and bottom plots. The open boundary conditions in the forced city simulation are also allowing the wind to flow outward without artifacts. As expected, the strongly stable ABL inversion remains intact in the precursor simulation, while it is mixed in the forced city simulation due to mechanically generated turbulence from the tall buildings, leading to gravity wave propagation in the stable layer above the inversion. Also, there is no evidence of artifacts arising from the stretched vertical grid in either the precursor or the forced simulations.

Figure plots instantaneous potential temperature at t=10 h for the concurrent turbulent precursor simulation (left) and the forced simulation with the city included (right) at y=1676 m. From here, again, it is clear that the stable inversion remains intact in the precursor simulation and is mixed by turbulence generated by the buildings in the forced city simulation. Further, warmer potential temperature mixed downward from the stable boundary layer leads to slight warming in the leeward side of buildings where the vertical velocity is prone to descent.

Figure plots the time-average over t∈[8,10] h of u-velocity (left), v-velocity (middle), and potential temperature (right) averaged over the city domain in the horizontal for precursor simulation and the city simulation forced by the precursor. From here, it is clear that the presence of the city leads to a reduction in wind speed in the atmospheric boundary layer as well as an increase in the potential temperature inversion height due to blocking and increased turbulence from the tall buildings. There is no indication of artifacts in these plots over the z∈480,600 m grid spacing transition region as well.

To get a sense of the performance of portUrb on GPUs compared to a similar model, the convective ABL simulation is performed with 2048 × 2048 × 122 grid cells in the x, y, and z directions, respectively with a grid spacing of 20 m in all directions. The FastEddy model reports a performance of 7.3 time steps per wallclock second in using 32 Nvidia V100 GPUs. Since access to this model of GPU is no longer available to the current authors, portUrb is run on Nvidia A100 GPUs and AMD MI250X GPUs using a number of GPUs that roughly equates to the same aggregate memory bandwidth. One could also make the choice to equate the aggregate compute throughput in terms of floating point operations per second (flop s⁻¹), but most GPU kernels in weather, climate, and LES models have relatively low arithmetic intensity. While the reconstruction kernels in portUrb are arithmetically intense, most other routines have the typical low arithmetic intensity of stencil-based fluids codes.

Each MI250X GPU is composed of two Graphics Compute Dies (GCDs), and therefore, the values for AMD MI250X GCDs will be used. The peak memory bandwidths of an Nvidia V100 GPU, an Nvidia A100 GPU, and an AMD MI250X GCD are 900, 1555, and 1638.4 GB s⁻¹, respectively. While this would suggest using 18 A100 GPUs and 18 AMD MI250X GPUs to give the aggregate memory bandwidth of 32 Nvidia V100 GPUs, an MPI decomposition of 3×6 tasks is less advantageous in terms of parallel data transfers than an MPI decomposition of 4×4 tasks. Therefore, 16 GPUs are used as a comparison point in this study, even though the aggregate memory bandwidths are only 85 %–90 % of the 32 V100 GPUs, putting this study's results at a slight memory bandwidth disadvantage. GPU-aware MPI is used in all simulations so that GPUs on separate MPI tasks are communicating directly from GPU memory rather than moving data to and from host memory. For sake of readability and simplicity, no effort is made in portUrb to use constant memory, texture caches, or user-controlled L1 cache.

The performance of portUrb in terms of time steps per wallclock second for different model configurations (order of accuracy and use of a Weighted Essentially Non-Oscillatory limiter) are given in Table . The fifth-order-accurate configuration without the WENO limiter is likely the most comparable to the FastEddy code and gives comparable performance on Nvidia A100 GPUs using similar aggregate memory bandwidth. On A100 GPUs, 9th-order accuracy is 36 % more expensive than 5th-order accuracy without the WENO limiter and 53 % more expensive with the WENO limiter. The AMD MI250X GCD is generally about 40 % slower than the A100 using the same number of GCDs, even though the MI250X GCD has more memory bandwidth. This is likely due to the AMD HIP compiler showing higher register usage per kernel and having a small hardware managed L1 cache.