FEOTS is used for offline tracer simulations using transport operators diagnosed from a comprehensive ocean model, the “parent model”. Here we apply FEOTS to the Parallel Ocean Program, in the framework of the E3SMv0-HiLAT climate model . Our specific configuration is described in and is referred to as E3SMv0-HiLAT03. E3SMv0-HiLAT03 has a tripole grid with nominal 0.3∘ spatial resolution. The grid has 1200×800 grid cells and 100 levels in the vertical. The grid has a “seam” in the Arctic that connects the poles in Siberia and Canada. Although technically not eddy-resolving in most of the world ocean , this configuration has a vigorous eddy field and a realistic representation of the eddy-driven Zapiola Anticyclone in the Argentine Basin .

With O(108) df's, and an eddying field that requires transport operators at high temporal frequency (e.g., 5-daily averages), the data volume of the diagnosed IRFs can become unmanageable quickly. It is therefore critical to keep the number of IRFs to an absolute minimum. This means that we need to choose advection and diffusion treatments that have the most compact stencils. Typically high-resolution ocean models use a bi-harmonic mixing scheme to dampen out the dispersive errors caused by the advection operator e.g.,. Bi-harmonic mixing is clearly undesirable for our application, given its huge stencil. Of the three advection schemes currently implemented in POP , the centered and third-order upwind schemes require explicit diffusion to manage the dispersion error. The flux-limited Lax–Wendroff scheme does not, making this the only reasonable choice for our application, even though its stencil is quite large (27 grid points, 3×3×3) compared with the other two (7 grid points for centered, 13 for the third-order upwind scheme). This choice requires 53 impulse functions to capture the advection operator (compared to 34 impulse functions for the third-order upwind scheme) but eliminates the need for explicit diffusion.

The model is forced by the normal-year Coordinated Ocean-Ice Reference Experiments version 2 (CORE-II; ) climatology, which has been a widely used framework to force ocean and/or sea ice models for hindcast simulations. With a time step of 7 min the model typically yields maximum CFL values of O(10-1) or smaller. Although the model was run for 186 years, we diagnosed the transport operators for the 5-year period starting at simulation year 64. Even though 63 years of spin-up is not sufficient to fully equilibrate the stratification in the deep ocean, the main circulation features (e.g., boundary currents, the eddy field, the Zapiola Anticyclone) are well established by then, making this an appropriate data set to demonstrate the capability of FEOTS. We refer to for evaluation of the hydrography and circulation in the Argentine Basin in a companion simulation.

We model a passive dye tracer as a concentration field that is subjected to advection and diffusion, 1∂∂t(1+v)c+∇⋅uc-K∇c=0, where v is the fluid volume anomaly, c is the tracer concentration, t is time, u is the ocean velocity field, and K is the diffusivity tensor that models unresolved eddy activity. The fluid volume anomaly is a unitless quantity that is a measure of the relative change of the fluid volume due to movement of the free surface. This formulation is chosen so that the total tracer is conserved and so that the offline model is consistent with the parent model .

In practice, the fluid volume anomaly is defined as the ratio of the free-surface height to the uppermost grid cell; only in the uppermost grid cell 2v=ηdz1δk,1, where η is the free-surface height, dz1 is the grid cell thickness in the uppermost grid cell, and δk,1 is the Kronecker delta function.

The initial and boundary conditions are set to be 3c(t=0)=c0(z,θ,ϕ) and 4c=cb(z,θ,ϕ,t), where z is depth (measured positive downward), θ is longitude, and ϕ is latitude. The semi-discrete form of Eq. () can be written as 5∂∂t(1+v)c+A+Dh+Dvc=0, where c is a vector of the discrete values of the tracer, A is the advection matrix, Dh is the horizontal diffusion matrix, and Dv is the vertical diffusion matrix. In the examples presented in this paper, the advection matrix corresponds to the flux-limited Lax–Wendroff advection scheme on an Arakawa B grid; this scheme is chosen in the parent model configuration. Since we do not enable explicit lateral tracer diffusion in the parent model in this study, all elements of Dh are zero. To create the vertical diffusion matrix, we diagnose diffusivity coefficients, which are generated from the KPP parameterization, and create a vertical mixing operator offline using centered differencing with no-flux conditions applied at the ocean free surface and at the seafloor.

The purpose of capturing the impulse response functions is to diagnose sparse matrices that are consistent with the advection discretization in the parent model. To illustrate this procedure, suppose that the parent model has N grid cells and that N impulse fields are set as the Kronecker delta functions: 6[c(i)]i=1N=[δj,i]i=1N. In this setup, impulse function i is zero at all grid cells except for grid cell i where the impulse function has a value of one. Application of the transport matrix to each of the c(i) returns column i of A, 7Ac(i)=∑j=1NAm,jδj,i=Am,i for m=1,2,3,…,N. While using a set of Kronecker delta functions will completely diagnose all of the elements of the transport matrix, this strategy is computationally expensive. For each time step this strategy requires computing the advective tendency for N tracer fields, where N is the number of grid cells. For example, coarse-resolution models at O(1∘) resolution have roughly 106 grid cells. Storing 106 impulse and impulse response functions would require approximately 3 TB of memory at double precision.

To reduce the number of required impulse functions to fully diagnose the transport matrices, we can take advantage of the fact that the advection scheme results in a sparse matrix. Equivalently, the domain of influence of the advection operator is limited to nearby grid cells. The parent model employed in used a third-order upwind scheme, where the impulse response is guaranteed to extend no further than two grid cells in each spatial dimension, giving a 5×5×5 brick for the domain of influence. Because of this, the authors used a set of 125 tracer fields, 8c(i,j,k;i0,j0,k0)=δi0,i(mod 5)δj0,j(mod 5)δk0,k(mod 5) for i0=1,…,5;j0=1,…,5;k0=1,…,5. FEOTS offers a unique capability to generate a minimal set of impulse functions by posing the problem as a graph-coloring problem. A graph G(V,E) is defined by a set of vertices V and edges E that connect the vertices. Two vertices connected by an edge are said to be adjacent. A valid graph coloring of G(V,E) assigns colors to each vertex so that no two adjacent vertices have the same color. To calculate impulse functions that can be used to diagnose transport operators, FEOTS offers functionality to express a POP mesh and an advection stencil into an equivalent graph that is colored with a greedy algorithm. This formulation has the benefit that it can be generalized to parent models based on unstructured grids, and it takes into account irregular boundaries from variable bathymetry.

In FEOTS, graph vertices V correspond to each ocean grid cell, centered on tracer points, in the POP mesh. Two vertices are adjacent if their impulse response functions overlap. Because a valid coloring results in adjacent vertices having distinct colors, vertices with the same color can safely be assigned to the same impulse function. Consequently, the chromatic number of the graph corresponds to the number of impulse functions used for model diagnosis. For this work the parent model uses a 0.3∘ periodic tripole mesh and the third-order flux-limited Lax–Wendroff advection scheme. This approach results in 53 impulse functions required to uniquely diagnose the transport operators.

The transport operators in Eq. () are diagnosed empirically from the parent model, using the methodology used by and pioneered by . This process involves diagnosing and time averaging the impulse response functions corresponding to each of the 53 impulse fields in the passive tracer equations in POP. Modifications are made in POP to initialize the passive tracers to the impulse functions at the beginning of each time step. The impulse response function is set equal to the advective tendency as diagnosed in POP. After diagnosing the IRF in each time step, the tracer field is reset to the impulse field so that the IRF at the next time step is saved. The IRFs are time-averaged over a configurable averaging period and written out to file.

For each time-averaged IRF, we create a sparse matrix representation of the advection operator in POP. Modeling the advection operator as a matrix–vector multiplication, Eq. () shows that passing an impulse at grid point i through the advection operator returns column i of the matrix that corresponds to the advection. We use this to construct a sparse matrix in compressed-row-storage format that corresponds to the advection operator.

For our test problems, we diagnosed the 5 d averaged IRFs and vertical diffusivities for the 5-year analysis period of the parent model. We repeated the simulation for 105 d, diagnosing 1 d averaged IRFs. With this methodology and the 7 min time step, the 1 d averaged operators are each an average of 1440 IRF snapshots, and the 5 d averaged operators are each an average of 7200 IRF snapshots. The data volume of the global parent model's 5 years' worth of 5 d averaged operators (365 IRFs and diffusivities) is about 9 TB. Once transformed to transport operators, the data volume is 4 TB.

The parent model uses a non-linear flux-limited Lax–Wendroff advection scheme; the flux limiter is equivalent to the ULTIMATE flux limiter described in and . Because FEOTS treats offline advection as a linear operation, online advection with the parent model, in this case, can not be equivalent. In what follows, we illustrate how the impulse functions result in the diagnosis of a more diffusive advection scheme than Lax–Wendroff.

In POP, the tracer equations are discretized using a finite volume approximation on an Arakawa B grid. The non-linear flux-limited Lax–Wendroff scheme is applied in three dimensions using a dimensional splitting technique, the details of which can be found in Chap. 6 of . For the purpose of this discussion, it is sufficient to consider the one-dimensional problem of calculating the advective flux at a tracer cell face. The advective flux at a tracer cell face can be written as 9Ff=ufcu+Ψ(r)(cd-cu), where Ff is the advective flux at a tracer cell face, uf is the normal velocity at a tracer cell face, cu corresponds to the tracer concentration in the upwind direction, cd corresponds to the tracer concentration in the downwind direction, and Ψ(r) is the limiter function where r is a measure of the monotonicity of c. The monotonicity in this method is defined as 10r=cu-cu-1cd-cu, where cu-1 is the tracer concentration one cell further in the upstream direction. When r<0, the tracer concentration has a local maximum or minimum centered around cu. For the ULTIMATE flux limiter, Ψ(r)=0 when r<0, in which case the flux evaluates to the upwind flux.

The impulse functions used in this study all have a local maximum centered at the impulse locations. At the impulse locations r<0, and therefore the diagnosed flux is the more diffusive upwind flux. This behavior will be apparent and quantified when comparing the online and offline simulations in Sect. .

Forward integration of the offline tracer model uses a backward Euler method for vertical mixing and can use forward Euler, Adams–Bashforth second order, or Adams–Bashforth third-order methods for transport. As in , we forward step an equation for the volume anomaly using a forward Euler method. Volume anomalies arise due to divergence in the transport field at the uppermost z level that are associated with fluctuations of the free surface.

In general, the time integration scheme can be written as 11vn+1=vn+ΔtAni,12(I+Vn+1+Dv)cn+1=(I+Vn)cn+Δt(A+Dh)c*, where v is the volume anomaly, i is a constant vector whose elements are all set to one, c* depends on the time integration scheme that is used (Table ), and Vn+1 is a diagonal matrix whose diagonal elements are the volume anomalies.

In ocean models, advection and horizontal diffusion operators have a compact stencil, enabling the use of the IRF approach described in the previous section to “capture” these advection and diffusion operators. However, vertical diffusion is usually treated differently, by solving a tridiagonal system that touches the entire water column. The reason is that high values of vertical diffusivity are applied where the water column is unstable, and these values easily render any explicit scheme unstable. Consequently, as the region of influence of the vertical diffusion operator is the entire water column, the IRF approach would demand a separate IRF field for each vertical level of the model grid, which is prohibitive for finely resolved grids. Instead, FEOTS treats the vertical solve similarly as the parent model, so rather than IRFs, the vertical diffusivities are diagnosed, saved, and used to recreate the vertical diffusion operators offline.

Forward-stepping FEOTS requires inverting a tridiagonal system of equations, given by Eq. (), for the tracer concentration in order to incorporate vertical mixing. To solve this system, we use the preconditioned conjugate gradient algorithm with a diagonal preconditioner. The initial solution guess for the vertical mixing solver is set as the tracer concentration that is predicted without vertical mixing, 13c0n+1=(I+Vn+1)-1((I+Vn)cn+Δt(A+Dh)c*).

For the results presented in this paper, we use the third-order Adams–Bashforth time integrator, and the conjugate gradient solver is stopped when the residual magnitude, relative to the initial solution guess magnitude, is less than 10-6. We use a 15 min time step, and a typical maximum CFL value, obtained by eigenvalue analysis of the transport operators, is O(0.1).

We now show that constant preservation is expected in the discrete system. This provides the foundation for a test case that we use to verify the FEOTS implementation. From Eqs. () and (), we start by assuming that there is no lateral or vertical diffusion, and the initial tracer field is a constant value of 1 (c0=i). The discrete system at the initial time step is then 14av1=v0+ΔtAi,14b(I+V1)c1=(I+V0)i+ΔtAi.

Note that we can write these equations using indicial notation, which makes it easier to obtain the result for c1: 15avj1=vj0+Δt∑kAj,k,15bcj1+vj1cj1=1+vj0+Δt∑kAj,k.

Substituting Eq. () into () gives 161+vj0+Δt∑kAj,k]cj1=1+vj0+Δt∑kAj,k.

The only solution to Eq. () is cj1=1 for each j, implying that the discrete solution remains a constant. If we progress to the next time step using a forward Euler method, the same result is obtained. If instead we switch to a second-order Adams–Bashforth method for the next time step, we have 17ck*=3ck1-ck02=1. Using this for c* results in 181+vj2+Δt∑kAj,kcj2=1+vj1+Δt∑kAj,k, which again implies that c2=i. This result also holds for the third-order Adams–Bashforth method in this case, since 19ck*=23c2-16c1+5c012=1 when c2=c1=c0=i.

When vertical mixing is introduced, the solution obtained in Eq. () serves as an initial guess to the conjugate gradient solver. If the initial guess for the solver is a constant tracer field, 20Dvc=0. This result in Eq. () is due to the use of a finite volume discretization for the vertical diffusion with no-flux conditions at the ocean surface and at the seafloor. In exact arithmetic, we would therefore expect the tracer field to remain a constant.

This analysis shows that a constant tracer field remains a constant tracer field under the discretizations employed in FEOTS. Finite precision arithmetic, however, can produce slight deviations from a constant field, and the results presented in Sect.  characterize the behavior of round-off errors for a constant tracer field, both with and without vertical mixing.

As discussed in the introduction, the Argentine Basin is an ideal region for testing a tracer transport capability in an eddying ocean model. In this paper we compare online and offline simulations of dye tracers that are initialized at the boundaries of the Argentine Basin, here chosen as (52.18∘ S, 28.06∘ S) × (70.25∘ W, 24.90∘ W). In all simulations, the only source of tracers comes from the model boundary conditions applied along the southern, eastern, and northern boundaries.

We want to distinguish between water masses that originate from each of the domain boundaries and above and below 1000 m depth. This is accomplished by simulating six passive tracer fields Di with boundary conditions: 21aD1:cb1=H(1000-z)when ϕ=52.18∘S,21bD2:cb2=H(z-1000)when ϕ=52.18∘S,21cD3:cb3=H(1000-z)when θ=24.90∘W,21dD4:cb4=H(z-1000)when θ=24.90∘W,21eD5:cb5=H(1000-z)when ϕ=28.06∘S,21fD6:cb6=H(z-1000)when ϕ=28.06∘S, where H(x) is the Heaviside step function. With this configuration, tracers D1, D3, and D5 are released in the upper 1000 m on the southern, eastern, and northern boundaries, respectively; while tracers D2, D4, and D6 are released below 1000 m. Note that maximum mixed-layer depths in the Argentine Basin in winter are around 500 m in this model, so deep convection should not play a role in the transport of these tracers across the 1000 m depth horizon.

The parent model uses a finite volume discretization that guarantees the preservation of constant tracer fields. To verify that FEOTS accurately diagnoses transport operators that are representative of the parent model, our first simulation involves verifying that a constant tracer field remains constant under the action of the diagnosed transport operators.

In all of our simulations, we have opted to use single precision arithmetic and have enabled aggressive compiler optimizations (compiler option -Ofast with GCC 9.2.0). These choices were made to minimize data storage costs for the transport operators and post-processing output and to optimize the time-to-solution for the offline simulations. Although analytically we expect that the FEOTS algorithm should preserve constant tracer fields, errors from floating point arithmetic are expected to be the main source of constant preservation errors.

To better understand the spatial distribution and the relative impact of round-off errors, we configure a simulation where the initial tracer field is set to c0=1, and there is no external source or sink of tracer. Since ∇c0=0 and ∇⋅u=0 under the discretizations used in the parent model, we expect that ct=0 and c=c0=1 for all time, as demonstrated in Sect. .

Figure  shows the maximum volume anomaly in the domain as a function of depth after 5 d and 5 years of integration. Analytically, the volume anomaly is expected to be zero in all cells except the topmost layer. At the surface layer, fluctuations in the free-surface height are associated with non-zero fluid divergences that contribute to changes in the fluid volume. Beneath the surface layer, the fluid velocity field is expected to be divergence free. In general, larger errors in the volume anomaly are observed above 1000 m depth. After 5 d, errors in the deep ocean are O(10-5), and after 5 years, the deep ocean volume anomaly errors have grown by an order of magnitude to O(10-4). Larger errors are observed above 1000 m, reaching O(10-3) after 5 years. Note that the volume anomaly field is identical for all choices of the time integrator for the dye tracer and is independent of vertical mixing.

Errors in the volume anomaly lead to spurious values for predicted tracer concentrations. For this simulation, any deviation of the tracer concentration from its initial uniform value is erroneous. Figure  shows the max error in the dye tracer as a function of depth after 5 d and 5 years of integration, with and without mixing. After 5 years of integration, the maximum relative error with mixing is about 0.05%, and without mixing it is about 0.01%. At depth, the errors in the tracer are comparable with and without mixing. However, above 1000 m, particularly in the mixed layer, the inclusion of mixing results in an accumulation of round-off errors.

In addition to quantifying errors for the constant tracer scenario, a practical concern is in the comparison between the online and offline tracer simulations. Do the tracer distributions simulated by FEOTS using transport operators diagnosed from the parent model E3SMv0-HiLAT03 (offline) faithfully represent the tracer distributions simulated by the parent model itself (online)? The time averaging of the transport operators will introduce differences between the online and offline simulations, as will the flux-corrected advection operator. Our aim is to quantify and qualitatively describe the differences between a tracer simulation conducted directly in the online parent model and the offline model. We do this for our example problem, namely determining the source waters of the Zapiola Anticyclone.

Figure  compares the tracer fields obtained with the online and offline method, at the end of the 5-year analysis period. The dye tracers shown are those that are released in the upper 1000 m at each of the three domain boundaries. Visual inspection shows that the online simulation has sharper gradients compared to the offline simulation, even though the same dynamical features are visible. This is a result of the non-linear advection operator that is more diffusive when applied to impulse functions than to a smooth tracer field, as explained in Sect. .

The advective errors also affect integrated quantities of dye tracers within the Zapiola Anticyclone. Figure  shows the vertical distribution of tracer concentrations averaged over the Zapiola Anticyclone, while Fig.  shows the total tracer stock.

Dye tracer D1, sourced at the southern boundary in the upper 1000 m, reaches the ZA after about 45 d in the online simulation (Fig. , upper row). Concentrations in the upper 1000 m rise steadily in the first 2 years, then become mostly steady, indicating saturation is reached (Fig. a, black dashed). Just below the surface, concentrations in the final 3 years average about 0.76, suggesting that 76 % of the surface waters within the ZA may be derived from the Southern Ocean through the Malvinas Current. At 984 m this fraction is 0.45. Tracer stock below 1000 m increases slowly and contains 17 % of the column stock after 5 years (Fig. a, black dotted). The offline simulation reproduces this behavior qualitatively but with significant quantitative differences. D1 reaches the ZA about 5 d earlier and increases faster than the online simulation, but upper-layer stock plateaus at a slightly lower level (Fig. a, gray dashed). Tracer fraction reaches 0.74 just below the surface, close to the online simulation, but the saturation value of 0.37 at 1000 m depth is significantly lower. Figure  (upper row) shows that this is due to a significant vertical redistribution of tracers by vertical diffusion that depletes tracers from the upper 1000 m and increases concentrations below. Indeed, the inventory below 1000 m accounts for 30 % of the column stock after 5 years.

The story is similar for dye tracer D2, which is sourced at the southern boundary below 1000 m. It arrives in the ZA after 87 d in the online simulation. The stock below 1000 m (Fig. b, black dotted) rises quickly through the first 2 years, followed by a more gradual (linear) rise after that. Tracer concentrations at 1049 m level out at 0.20 after 3 years. The total contribution of Southern Ocean waters (D1+D2) is about 59 % at 1000 m depth. The offline simulation reproduces this behavior quite well but displays higher inventories in the upper 1000 m that increase the overall column stock by 9 % after 5 years. The vertical profiles again clearly show the impact of vertical diffusion that depletes tracers in the 1000–3000 m depth range and increases concentrations below and above (Fig. , second row). The D2 concentration at 1049 m depth levels out at 0.21, with a total Southern Ocean contribution at 1000 m of 56 %.

The next-largest contribution to the ZA tracer inventory is coming from the north through dye tracers D5 (upper 1000 m) and D6 (below 1000 m). It takes about 110 d for D5 to arrive at the ZA, and the surface concentration saturates at about 0.18 after 550 d (as does the upper-layer tracer stock; Fig. e, black dashed, overlain by solid). This suggests that the Brazil Current may contribute about 20 % of the surface waters in the ZA. At 984 m, however, this fraction is still only 0.015 after 5 years, and rising, probably reflecting the strongly sheared character of the Brazil Current and the long transit time from the northern domain boundary to the ZA. Notable concentrations of D6 reach the ZA after 2 years, but trace quantities already arrive after about 270 d, having been mixed upward into the upper layer and transported southward in the Brazil Current. The offline simulations display qualitatively similar behavior, but D5 inventories are significantly higher (+40%), with again significant sequestration below 1000 m. D6 stock is slightly lower (-7%) than in the online simulations, despite higher values above 1000 m.

Dye tracers D3 and D4, released at the eastern domain boundary, take much longer to reach the ZA due to very low westward flow velocities in the interior part of the basin. Offline stock of D3 in the upper 1000 m is about 50 % smaller than in the online simulation, a deficiency that can only be partly explained by sequestration below 1000 m. D4 stock is 65 % too high, with enhanced stock both below and above 1000 m depth.

Based on the propagation speed of the diffusion front of D1 in the offline simulation (Fig. a, center column), we can make a rough estimate of the artificial vertical diffusivity that is introduced by the advection issue. We are able to model the depth of the diffusion front below z0=-1000 m and after t0=132 d as z=z0-4D(t-t0), when D=1.74×10-2 m2 s-1. This is 2 to 3 orders of magnitude larger than typical values for background diffusivity used in ocean models.

The parent model is capable of producing velocity fields that have a wide range of scales of spatial and temporal variability. The shortest temporal periods are on the order of a few time steps, and the longest period is the duration of the simulation. In general, higher-resolution models introduce more variability on shorter length and time scales, and some consideration is needed when selecting an averaging period for the transport operator diagnosis. For storage reasons, it is not practical to store snapshots of the transport operators at every time step. Conversely, representing the ocean transport with long time averages may exclude the effects of important variability. The choice of the time-averaging period for the transport operators can impact the evolution of tracers calculated in FEOTS, and an appropriate balance of practicality and accuracy should be struck.

To our knowledge, there is currently no definitive guidance on choosing the averaging period for velocity fields in an offline tracer simulation; the averaging period should be chosen so that variability in the underlying advection field is well resolved. Here, the impact of temporal averaging is investigated by comparing simulations with 1 and 5 d averaged operators. To that end, we ran another online simulation for which we saved 1 d averaged IRFs. This simulation was run for 105 d, producing 105 operator sets. The offline simulations were run for 365 d hence cycling through the operator set almost 3.5 times.

The online simulations that produced the 1 and 5 d averaged operators were not bit-for-bit identical, as we inadvertently specified different processor counts. A positive consequence of this oversight is that we have two different realizations of the same chaotic system, allowing us to assess how the tracer advection error compares to trajectory divergence, in terms of their impact on tracer stock. Figure  shows that averaging 1 d averaged operators to 5 d averages has a very small impact on the tracer stock (light gray); the impact on spatial distributions is similarly small (not shown). This shows that 5 d averaged operators are sufficient to accurately reproduce tracer distributions. Comparison of the black and dark gray lines shows that the impact of trajectory divergence on tracer stock (dark gray) is of similar magnitude as the impact of the advection error (black), at least for the first year of simulations.

One goal of FEOTS is to perform tracer calculations at a lower computational cost than the parent model. Additionally, FEOTS allows researchers to take advantage of transport operators produced by state-of-the-art climate simulations to conduct regional offline simulations. This provides flexibility in studying ocean transport phenomena and increases the value of online produced model data while considerably reducing the computational expense for researchers solely interested in studies involving passive tracers. Here we evaluate the computational performance of a regional FEOTS configuration and compare it with the global parent model.

The total cost of using FEOTS is associated with the following steps:

impulse functions are generated from the model grid,

Table  summarizes a comparison of the computational expense and compute platform size requirements between POP and FEOTS. We ran the 0.3∘ ocean/sea ice configuration of E3SM-HiLAT on the Los Alamos National Laboratory's Institutional Computing clusters. A typical simulation with six dye tracers costs 9020 CPU h per simulated year, using 2432 cores on 76 nodes. The throughput is 6.5 simulated years per wall-clock day. The simulation with 53 IRFs (in addition to the six dye tracers) typically costs 45 000 CPU h, with a throughput of 1.60 simulated years per day when using 2912 cores.

In contrast, a 1-year offline FEOTS simulation of the Argentine Basin problem with six tracers takes roughly 47 CPU h and a throughput of 3 simulated years per day on six cores. Table  shows the five most expensive routines in FEOTS and the percentage of the total wall time spent executing those routines. The iterative treatment of vertical mixing is most expensive; a 1-year simulation without vertical mixing costs 6.8 CPU h and provides a throughput of 20 simulated years per day on six cores. All of these routines execute sparse matrix–vector multiplication in order to compute advective and diffusive tendencies, suggesting future improvements to sparse matrix–vector multiplication in FEOTS would be beneficial.

Currently, offline global simulations at eddy-resolving resolutions are slow and require a large amount of memory per core. For our global model, about 15 GB of memory is needed for the offline simulation at single precision and 30 GB at double precision. At the time of this study, our focus was on proving the transient simulation capabilities, which could be done at a low computational cost in regional simulations; we did not obtain direct measurements of FEOTS' runtime for global transient simulations. However, we are interested in understanding whether this methodology is potentially computationally competitive in comparison to online tracer simulations in the parent model.

To develop an estimate for a global offline simulation with FEOTS, we assume that the CPU hours scale linearly with the number of grid cells. This is reasonable, since the majority of FEOTS' runtime is spent in sparse matrix–vector multiplication, which scales linearly with the number of rows. The Argentine Basin simulation has 1.02×106 grid cells, whereas a global configuration has about 45 times more grid cells. Thus, an offline simulation with FEOTS is expected to cost about 2115 CPU h per simulation year in the current version, 77 % less CPU hours than the parent model. However, the expected throughput for an offline global simulation is estimated as 1 model year for every 15 d. Although FEOTS is currently expected to take longer to produce global transient simulations, it is probable that exposing parallelism in FEOTS will provide comparable runtimes to the parent model at a reduced computational expense. Though this is not a definitive result, it is a reasonable estimate that motivates future work in exposing parallelism in FEOTS.