Performance of offline passive tracer advection in ROMS (v3.6, revision 904)

Offline advection schemes allow for low computational cost simulations using existing model output. This study presents the approach and assessment for passive offline tracer advection within the Regional Ocean Modeling System (ROMS). An advantage of running the code within ROMS itself is consistency in the numerics on and offline. We find that the offline tracer model is robust: after about 14 days of simulation (almost 60 advection timescales), the skill score comparing offline output to the online simulation using the TS_U3HADVECTION and TS_C4VADVECTION (3rd-order upstream horizontal ad5 vection and 4th-order centered vertical advection) tracer advection schemes is 99.6% accurate for an offline time step 20 times larger than online, and online output saved with a period below the advection timescale. For tracer advection scheme MPDATA, accuracy is more variable with offline time step and forcing input frequency choices, but is still over 99% for many reasonable choices. Both schemes are conservative. Important factors for maintaining high offline accuracy are: outputting from the online simulation often enough to resolve the advection timescale, forcing offline using realistic vertical salinity diffusivity values 10 from the online simulation, and using double precision to save results.


Introduction
The ability to integrate Eulerian tracer fields offline, or separate from the online, original full simulation is attractive because of the time savings. Once an online simulation has been run, any number of offline simulations can be run, forced by the Previous work has been done in this area with other models. An offline tracer model for the MIT general circulation model (MITgcm) was developed and showed good accuracy (Hill et al., 2004). An offline tracer model based on MITgcm has been used in several studies (Dutkiewicz et al., 2001;McKinley et al., 2004). Another offline tracer model, OFFTRAC, is based on the Hallberg Isopycnal Model (HIM) and has been used for long-term biogeochemical integration (Zhang et al., 2014). 25 Other tracer models have been developed separately from a full numerical ocean simulator. Gillibrand and Herzfeld (2016) have developed a separate tracer advection model that is not numerically limited by the Courant number as is expected in the present case. Another such model developed by Khatiwala et al. (2005) has a different approach entirely to offline tracer advection, using a mathematical approach that is distinct from more commonly used numerical tracer integration. Interestingly, Lévy et al. (2012) found that for particular dynamical scenarios, degrading online model output spatially can result in offline 30 computational savings with little accuracy degradation. The offline tracer model described in this paper is integrated into and derived from the ROMS model: preprocessor flag choices allow access to the offline capability. While not being derived from a specific ocean model allows for wider potential use, as in some of the previously-described models, there may be an advantage in using the offline model that is derivative of the offline model to ensure consistency, using the exact same numerics and setup. The expected user for this software is someone 35 who uses ROMS for their ocean modeling needs and wants to have the ability to run more tracer simulations, decoupled from their more expensive online simulations. Another type of user may simply have some ROMS output available, and this code will allow them to leverage it beyond its originally intended use.
The experimental setup is described in Section 2; this includes the description of the model setup (Section 2.1), the offline experiments (Section 2.2), and the metrics used for evaluation (Section 2.3). Results are shown in Section 3, and a discussion 40 of results is in Section 4. Specific code descriptions are in the Appendix: code changes made for robust offline tracer advection (Section B) and a description of how to set up both online and offline simulations for best results is given in Section C.  to 10 days at the 18th interior grid. The climatology used for the nudging process is also provided by the HYCOM output. This online model ran for 90 days, from April 20 to July 19, 2010, but a subset of 14 days are used for the present experiments.
A series of online and offline simulations were run to evaluate comparative performance of offline tracer advection. All numerical experiments presented in this paper were initialized with a discrete Gaussian blob of dye southwest of the Mississippi river delta in a regional model of the north Gulf of Mexico (see Figure 1). The blob of dye extended fully through the water column. Online simulations were run with two tracer advection schemes: MPDATA (both horizontal and vertical) and TS_U3HADVECTION (3rd-order upstream horizontal advection) and TS_C4VADVECTION (4th-order centered vertical advection) (shortened to U3C4 for the remainder of the paper). Additionally, the online simulations were output at different frequencies, as multiples of the time step (the nhis and navg parameters for ROMS): 2, 5, 10, 20, 50, 100, 200, 500, 1000, 2000, 5000. Given that the time step of the online simulation was 20s, these correspond to output frequencies of about 20s, 40s, 100s, 3.3min, 6.7min, 16.7min, 0.56h, 1.1h, 2.8h, 5.6h, 11h, and 28h. Online simulations were saved as both instantaneous snapshots (his files from ROMS) and as averages across time steps (avg files from ROMS).

60
The relevant controlling timescale for this simulation is the advection timescale. Results from online simulations of the dye advection show a representative length scale of about L = 10km and speed of about U = 0.5m/s, giving an advection timescale of T = L/U = 20000s, or about 5.6hr. This timescale is specific to the location of the dye patch, which is off the continental shelf and responding to mesoscale processes, as opposed to if the dye patch was on the shelf in which case one would expect a shorter timescale. The timescale will be used to normalize times given in results and to interpret accuracy in relation to offline

Offline experiments
Offline simulations were run using one of the two tracer advection schemes, with either his or avg files as climatology at the output frequency from the online simulations (controlled by nhis/navg). Additionally, they were optionally forced by the vertical salinity diffusion variable, Aks, as calculated by the online simulation or by just the background value. Finally, for an 70 input climatology file from an online simulation of a given output frequency (nhis/navg), offline simulations were run with a time step from the list of nhis values of up to the same output frequency as the online simulation. A time step of 50 times the online time step was found to lead to unstable solutions, so in effect, the offline time step could be 1, 2, 5, 10, or 20 times the online time step, but never larger than the nhis value for the given simulation. Also note that the offline time step needs to divide evenly into the output frequency in the climatology file so that only two time steps are being accessed at a time. So, 75 for a climatology forcing file of nhis=50, the offline simulation could not be forced with dt=20.

Metrics: skill score
The main metric used to evaluate the performance of this model is a skill score, SS (Bogden et al., 1996;Hill et al., 2004;Hetland, 2006). This is calculated as:

80
where D off and D on are the volume of dye on the 3D grid and in time for the off and online simulations, and the brackets .
indicate averaging over horizontal and vertical dimensions, returning a time series.
Often skill scores are calculated with respect to a reference. For example, for numerical model performance, the difference between model and data in the numerator may be compared with the difference between climatology and data in the denominator in order to assess how much better the model is performing than simple climatology (Hetland, 2006). An analogous 85 comparison may be made here versus persistence of the initial condition of the dye patch, so that this skill score shows how well the offline model performs compared to simply persisting the initial condition: Skill scores are a comparison between an offline simulation and the online simulation from which it is forced, unless otherwise noted, so that the skill score represents accuracy of the offline simulation to the online simulation, or the skill in faithfully 90 reproducing the online simulation. This is different from a measure of the accuracy of the online simulation itself to simulate the dynamics.

Simulations and software
Simulations were performed on a Linux cluster with 84 processors for online simulations and 28 for offline. The number used was not optimized.

Results
The accuracy of selected offline simulations are presented here. Since there were over 300 offline simulations, only selected nhis 500, about 2.8 hours), but has increasing impact for less frequent online model output (higher values of nhis, middle subplot). This means that for the present model setup and region, frequency of online output higher than about 1-3 hours is not important. Results are relatively similar with nhis 1000 (about 5 hours), but accuracy decreases significantly as nhis increases beyond that. Context for the nhis values is given in Section 4.
The importance of nhis and the offline time step together for tracer advection scheme MPDATA is shown in Figure 2   and whether the vertical salinity diffusion coefficient Aks is forced with the online simulation ("on") or a constant background number ("off"). These cases also have nhis=1 (online output was saved each time step) and dt=1 (offline time step matched online time step).
(Middle) Performance between MPDATA and U3C4 advection schemes with nhis values varying. These cases also have Aks forced from online and dt=1. (Bottom) Performance for varying nhis and dt parameters, where MPDATA is used and Aks is forced from the online case. All offline simulations here forced by his files.
6 https://doi.org/10.5194/gmd-2020-221 Preprint. Discussion started: 9 September 2020 c Author(s) 2020. CC BY 4.0 License. simply persisting the initial condition. Next is a demonstration of offline accuracy compared to online output when different tracer advection schemes are used (middle). For reference, simulations forced with the same tracer advection schemes both on and offline are shown as well (U3C4, black solid, and MPDATA, black dashed). We find a significant decrease in offline model accuracy when the offline advection scheme does not match the online scheme, because different numerical schemes have different numerical dispersion and diffusion properties leading to differences in tracer advection. For comparison, the "skill 130 score" comparing online U3C4 and online MPDATA output (gray dashed) is shown. The online-online comparison for the two schemes has comparable performance, though lower; it is not clear there is a reason that the on and offline combinations should be better or worse than this, but the issue was not further explored. The best fidelity to an online simulation will be found by forcing offline with the same tracer advection scheme as online. Also, forcing offline with a different tracer advection scheme from online will give results that are different from the online results on the order of the difference between the results of the 135 different tracer advection schemes themselves. Finally, the significant impact of using single precision output is demonstrated (bottom); it is best to save online model output for forcing offline simulations with double precision.
Several other issues were investigated but not plotted (they can be seen in the paper GitHub repository). Passive tracers are conserved in online ROMS simulations (Shchepetkin and McWilliams, 2005); offline simulations also conserve tracers. Only small differences were found between forcing offline simulations with snapshots (his files) or averages between time steps 140 (avg files) from online simulations. Finally, for simulations in which a realistic Aks field was not forced, the background value used for Aks was varied; we found this did not impact results.
An overview of results is shown in Figure 4. The objective of this figure is to display the competing factors -computation time (x-axis) and storage required (y-axis) -that will ultimately determine offline accuracy (colored markers). Skill scores are shown for four subsets of simulations: tracer advection scheme U3C4 with (diamonds) and without (downward-pointing trian-145 gles) Aks realistically forced, and tracer advection scheme MPDATA with (squares) and without (upward-pointing triangles) Aks realistically forced. The best compromise of storage, computational time, and skill score is where the skill score is still high -in one of the top classes, but with the lowest storage and time requirements. For the present set of simulations, this occurs for U3C4 with realistic Aks for nhis of 200 (more conservative) or 500 and dt of 20, and for MPDATA with realistic Aks for nhis 200 and dt of 5. Simulations in which Aks is not forced always have lower accuracy and the small storage 150 saving is probably not worth the loss, however, there may be circumstances in which online Aks is not available.

Discussion
The context of the performance difference found as a function of nhis values (Figure 2 (middle, bottom)) can be considered as the impact of loss of energy represented in the system (an approach also used by Qu and Hetland (2019)). For example, Figure 5 shows the power spectral density of the online simulation speed from near the middle of the dye patch. The output 155 frequency, nhis, from the online simulation controls how much energy of this spectrum the offline simulations receive, and therefore how much of the system's energy is represented offline. The amount of energy missing can be seen visually by the overlaid lines representing different output frequencies, nhis. Skill score results (Figure 2)     nhis values increase starting at nhis of 200 or 500 (about 1 to 3 hours), which correspond to between 1 and 5% of the total energy being lost to subsampling the output.

160
Comparing a relevant dynamical timescale to nhis is another way to provide context for its impact on offline accuracy.
A previous study evaluating an offline tracer from MITgcm model output found that for their global-scale model, the inertial period controlled the output rate necessary for robust results (Hill et al., 2004). We find an analogous result here, though the relevant timescale is the advection timescale (see Section 2.1). The advection timescale for this regional model is about 20,000s, which corresponds to an output rate from the online model of nhis=1000 times the online time step, which is indeed

Conclusions
This paper presents a description and evaluation of an offline tracer advection model developed within ROMS. The advantage of this is the ease and consistency with which ROMS users can use existing model output to force offline tracer simulations at low computational cost. The main approach of the offline model is to force variables zeta, u/v, and ubar/vbar from an 175 online simulation as climatology; normally climatology would be used in a ROMS simulation to nudge boundary conditions toward mean values, but in this case all grid cells are fully forced. Additionally forcing the vertical salinity diffusivity, Aks, improves model accuracy. It is also important that the online simulation output used to force the offline simulation have double precision.
We tested two tracer advection schemes, MPDATA and TS_U3HADVECTION with TS_C4VADVECTION (3rd-order up-180 stream horizontal advection and 4th-order centered vertical advection, called U3C4 here) in a regional simulation of the north Gulf of Mexico, and found that the offline simulations are able to reproduce online simulations to high accuracy. The most important control differentiating offline accuracy was the nhis parameter describing how often online simulation output was saved, as a multiple of the online time step, to be input into the offline simulation. For both tracer advection schemes and with Aks forced, the offline simulations showed high accuracy up to nhis=200 or 500, about 1.1 and 2.8 hours. This is consistent with requiring temporal information at a rate higher than the relevant dynamic timescale, in this case an advection timescale approximated here as approximately equivalent to an nhis value of 1000. The offline time step dt was not an important choice for offline simulations run with U3C4, as long as it was under about 50 (all had skill scores of 99.6% after 14 days).
However, for MPDATA offline simulations were highly accurate with a time step 5 times the online time step up to nhis=200, with some dependence on the offline time step. Overall, the results show that it is possible to get high fidelity results in offline  Code changes were made to avoid repeating processes offline that were already included online. Initialization is now minimal for offline simulations (ini_fields.F), and initial values are replaced by the first time step read in from climatology.
Updates to sea surface height zeta (calls to ini_zeta and set_zeta in main3d_offline.F) have been removed since the variable is directly forced in the offline simulation. Boundaries are not forced in the offline case (except for the 210 passive dye field): horizontal indices now start 1 earlier and end 1 later in each tile so that climatology is read into ghost cells in place of boundary conditions (set_data.F). The remaining processes are controlled through the user input file and pre-processor flags (Section C).
The offline simulation is missing much of the complex time stepping in an online ROMS simulation due to the missing numerics, leading to necessary code adjustments (set_data.F). Climatology for 3D variables (u/v, salt/temp, tke/gls) To fix a problem with reading in the climatology at the correct time step, a condition was added (get_2dfld.F and get_3dfld.F) that compares the differences in times to being less than half a time step, avoiding any problems with numerical precision.
240 omega, the mass flux perpendicular to the local s-coordinate, was already setup to be read in through climatology with the OCLIMATOLOGY flag, but results did not match on and offline. The lower vertical index in the call for omega in get_data.F was 1, which is used for rho grids instead of the w-grid omega is actually on, which starts at index 0.

Appendix C: How to set up simulations
Requirements and considerations for setting up online and offline simulations in ROMS or COAWST with the offline passive 245 tracer advection code are provided below.

C1.1 Input file
In the project input file (the * .in file, for example, https://github.com/kthyng/oil_03/blob/master/External/ocean_oil_03.in), the items below should be considered in addition to the typical input parameter selections:

250
-Choose whether to save output as snapshots at a single time or averages across time intervals (ROMS his vs avg files).
Your choice will be used to force the offline simulation. Present results show this choice does not significantly change results. We recommend using his files in the absence of any other preference since otherwise it is necessary to include the initial file prepended to the input avg file in CLMNAME.
-Output necessary variables for forcing the offline simulation. Variables zeta, ubar, vbar, u, v are required for forcing 255 the offline simulation, and Aks is optional for improved accuracy in the offline simulation (though it increases amount of storage required).
-Choose output frequency (parameter NHIS for his files or NAVG for avg files). This is how often ROMS will save output to a his or avg file, as a multiple of the time step, and in turn this is what will be used to force the offline simulation. Important considerations for this selection include acceptable simulation runtime and storage requirements.
260 Figure 4 gives a paradigm from which to decide this for simulations in general. In the present study, for U3C4 there was a drop in performance below an output frequency of 500 times the online time step, and below 200 or 500 times for External/varinfo-online.dat.

C1.2 Header file
In the project header file (the * .h file, for example, https://github.com/kthyng/oil_03/blob/master/Include/oil_03.h), the additional flags below should be considered: -Choose a tracer advection scheme. We tested two schemes and found both accurately reproduced the online results 270 offline, though U3C4 performed slightly better. Note, however, that online tracer advection performance itself depends on the dynamics involved; more information is available in Kalra et al. (2019). Also note that MPDATA requires more runtime than U3C4 (Figure 4).
-Use OUT_DOUBLE to output results with double precision to significantly improve your accuracy, though increase storage required (Figure 3).

C2.1 Input file
In the project input file (the * .in file, for example, https://github.com/kthyng/oil_off/blob/master/External/ocean_oil_offline. in), the items below should be considered in addition to the typical input parameter selections: -The output frequency (NHIS or NAVG) will not impact your offline simulation performance, but should be chosen to 280 well-represent the dynamics in your model.
-A reasonable choice for the offline simulation time step DT is a multiple of the online time step. Some testing for your model setup is warranted. The present study found that time step was not important for the U3C4 tracer scheme combination -a DT of 20 times the online time step gave as good of accuracy as the online time step itself. However, for MPDATA, only using the online time step gave the highest accuracy; for the next level down of accuracy a time step 285 of 10 times the online time step was adequate ( Figure 4). Note also that the offline time step needs to factor evenly into the online output frequency, and the offline time step cannot be larger than the online output frequency.
-All physics should be off in the offline case, except for anything directly impacting the offline tracer field (dye_01) itself, because it is included in the online output. This implies: -Boundaries should all be closed except for offline tracer fields (e.g., parameter LBC(isFsur)).

290
-Turn off river forcing and other sources or sinks that were forced in the online simulation.
-Do not force winds, bulk fluxes, etc, from the online simulation. lation, will be entirely enforced).
-Turn on flags for climatology forcing for sea surface height (LsshCLM), and 2D (Lm2CLM) and 3D momentum 295 (Lm3CLM). Turn on salt and temperature flags (LtracerCLM) if you want to read them in (see header section next).
-Only need to output sea surface height (zeta) and the offline dye(s) (dye_01) -other fields are best used directly from the online simulation (the vertical velocity w for example is not calculated properly in the offline simulation). The sea surface height is necessary to properly calculate tracer advection fluxes.

300
-Input as the climatology forcing (CLMNAME) the online model output. If forcing with an avg file from the online simulation, it is necessary to place a file containing the initial conditions first; this is possible by inputting a list of file names.
-For this offline simulation, point to file varinfo-offline.dat for VARNAME, which has been edited to include the new variables that can be input as climatology and so that all climatology time variables are named ocean_time. The 305 latter change allows for the online output to be input directly offline as climatology without processing the file to rename variable attributes. The file has been provided in the code repository: https://github.com/kthyng/COAWST-ROMS-OIL/ blob/master/ROMS/External/varinfo-offline.dat.

C2.2 Header file
In the project header file (the * .h file, for example, https://github.com/kthyng/oil_off/blob/master/Include/oil_offline.h), the 310 additional flags below should be considered: -Use the OFFLINE flag for any offline simulation, and additionally the OFFLINE_TPASSIVE flag for offline tracer advection.
-For best results, use the same tracer advection scheme as the online run. The schemes do not have to match but the skill score between the simulations will diminish substantially ( Figure 3) since they do not use the same numerics. We did 315 not test other tracer advection schemes but we have no reason to think they will not work offline.
-Forcing the vertical salinity diffusivity Aks as predicted by the online simulation gives better offline accuracy than not, though requires storing it from the online case. This can be forced with the AKSCLIMATOLOGY flag. More information on the offline flags is available in Section B.
Author contributions. KMT edited the code, performed final simulations and analysis, and wrote the text. DK, VRX, and LQ edited the