The hydrodynamic MARS3D (Model for Application at Regional Scale) model solves the primitive equations under the Boussinesq and hydrostatic assumptions. It uses finite differences over a staggered Arakawa C grid with a vertical generalized-sigma coordinate framework. It relies on a barotropic–baroclinic-mode splitting. The barotropic mode, which obeys the shallow-water equation systems, is treated with a modified alternating-direction implicit (ADI) algorithm (Leendertse and Gritton, 1971). Unlike split explicit free-surface models, the temporal integration of both barotropic and baroclinic modes is carried out with the same time step due to a wider stability range of the barotropic mode. The barotropic–baroclinic coupling is simplified, but the propagation of the barotropic mode is weaker as the scheme is implicit.

The original time scheme described in Lazure and Dumas (2008) for MARS3D v6 has been significantly modified. This is done, first, by adding a prediction step (assessment of depth-averaged velocity v‾n+1,∗) and, second, by alternating the sweep of the computational grid in a row–column-wise manner and then column–row-wise manner. These modifications have been introduced to limit the spatial split errors inherent in single time step schemes. The row–column-wise form is developed in Eq. (1). It consists of computing (depth-averaged velocity u‾n+1 and sea surface elevation ηn+1,∗) and then (v‾n+1ηn+1) in the first half of a time step. The second half of a time step, which symmetrically gives (v‾n+1ηn+1,∗) first and then gives (u‾n+1ηn+1) is not detailed. The first equation of the system is a local solver, whereas the second and the third equations solved together lead to a linear tridiagonal system whose size is twice the number of grid cells in a row because the unknown vector is made of (u‾n+1ηn+1,∗). Similarly, the fourth and fifth equations lead to a linear tridiagonal system whose size is twice the number of points in a column. 1v‾n+1,∗=v‾n-gΔt∂ηn∂y+ΔtGv(un-1vn-1)ηn+1,∗=ηn-Δt∂∂xhnu‾n+1+∂∂yhnv‾n+1,∗u‾n+1=u‾n-gΔt∂∂xαηn+1-αηn+1,∗+ΔtGuun,vnηn+1=ηn-Δt∂∂xhnu‾n+1+∂∂yhnv‾n+1v‾n+1=v‾n-gΔt∂∂xαηn+1-αηn+1+ΔtGvun,vn In Eq. (1), h stands for the total water depth, Δt is the time step and α is the implicit coefficient of the external pressure gradient term. The term Gu/v(unvn) gathers the vertical average of all the remaining terms, including the nonlinear and the horizontal dissipation terms, the Coriolis force and the friction at the surface and the bottom. All of these remaining terms are explicit and not discussed here.

The linear tridiagonal system is a non-local solver, where ηn+1,∗ and un+1 are solved over a whole row by the ADI algorithm. Then, for a massive parallel computation based on a decomposition domain made of horizontal tiles, ηn and un are distributed on the different tiles crossed by the considered row (Fig. 1). The linear system can be schematically written as diagonal blocks, where each block represents the linear tridiagonal system over a tile, as follows: 2coef  on  tile1⋯0⋮⋱⋮0⋯coef  on  tilenη,uon  tile1⋮η,uon  tilen=rhson  tile1⋮rhson  tilen. It shows that η and u need to be gathered into a single row tile and that, once the linear tridiagonal system is solved, the solution has to be broadcasted back to the original geographic tiles. In order to reduce the communications, the first three equations of the system (1) have been rearranged into the system (3) thanks to the splitting of un+1 into an explicit part and an implicit one. This leads to a tridiagonal system in which the unknown vector is made solely of ηin+1,∗. 3v‾n+1,∗=v‾n-gΔt∂ηn∂y+ΔtGv(un-1vn-1)u‾n+1,∗=u‾n-gΔtα∂ηn∂x+ΔtGu(unvn)-gΔt21-α∂∂xhn∂ηn+1,∗∂x+ηn+1,∗=ηn-Δt∂∂xhnu‾n+1,∗+∂∂yhnv‾n+1,∗u‾n+1=u‾n+1,∗-gΔt1-α∂ηn+1,∗∂x As a result, the length of the ADI system is now precisely the number of grid cells in the x direction for a given row, and the data involved in the message-passing interface between tiles are reduced by a factor of 2. The same methodology is applied to the fourth and fifth of system (1) in the y direction.

In addition to the modification of the barotropic solver itself, some additional modifications have been implemented in the barotropic–baroclinic coupling because the original iterative coupling described by Lazure and Dumas (2008) was not compatible anymore with the reformulation (3). The coupling that can be sketched (according to Fig. 2) is now straightforward. (a) The barotropic mode is forced with vertically integrated terms of advection, horizontal dissipation, Coriolis, internal pressure gradient and the bottom friction (assessed with the first level of velocity), (b) the baroclinic mode is forced with the external pressure gradient, and (c) the last classical step of the coupling consists of redistributing the vertical mismatch between the barotropic transport and the vertically integrated baroclinic transport in order to ensure mass conservation.

Last but not least, the wetting–drying capability has been modified to introduce partial drying or wetting. It consists of introducing a coefficient fwet to the non-local continuity equation that evolves in time, according to Eq. (4): 4ηn+1Sfwetn+1=ηnSfwetn-Δt∂xhnun+1Δy‾+∂yhnvn+1Δx‾, where S stands for the grid cell surface (S=ΔxΔy), and fwet is the time-dependent fraction of the grid cell that is flooded (0≤fweti,j≤1). fwet is given by a prognostic equation accounting for the bottom slope within the grid cell and the water column height at its center.

AGRIF (adaptive grid refinement in Fortran; Debreu et al., 2008) is a package for the integration of structured adaptive mesh refinement (SAMR) features within a multidimensional finite difference model. Its main objective is to simplify the integration of SAMR potentialities within an existing model, while making minimal changes. In particular, it includes a lexicographic analyzer of Fortran code that generates, at the compilation step, the data structures required for running the same code on any grid hierarchy. AGRIF is currently used in the following ocean models: ROMS-AGRIF (Debreu et al., 2012; Penven et al., 2006), a regional model developed jointly at Rutgers University and University of California, Los Angeles (UCLA), the NEMO ocean modeling system (Biastoch et al., 2018, 2009), a general circulation model used by the European scientific and operational communities, and HYCOM (HYbrid Coordinate Ocean Model), a regional model developed jointly by the University of Miami and the French Navy. In MARS3D, a one-way nesting implementation ensuring only mass conservation (Muller et al., 2009) was first introduced, followed by a two-way nesting for temperature and salinity only (Dufois et al., 2014). This paper aims to introduce a comprehensive two-way nesting in accordance with the improved MARS3D time scheme, along with its evaluation in coastal environments.

The MARS3D model and the AGRIF library can be run in sequential, open multiprocessing (OMP), message-passing interface (MPI) or hybrid (based on both MPI and OMP parallelization) modes. The OMP method is based on the classical fine-grain approach, where OMP directives share the work between threads, specifically at DO (in Fortran) directive loops. In order to avoid the overhead of work sharing, parallelization is done on the j-column loop, which is the inner index of the 3D variables in MARS3D. For the MPI mode, an optimized domain decomposition is defined before the run to balance the load on the different cores of the cluster. It consists of attributing approximately the same number of wet grid cells to each MPI rank and excluding the land-masked part of the domain. The hybrid computation combines the MPI (domain decomposition) and OMP (distribution of threads in Fortran loops inside each MPI domain) parallelizations. It has been implemented, following the MPI_THREAD_FUNNELED type, where MPI calls are only made outside OMP parallel regions.

For a MARS3D-AGRIF configuration with only one child per nesting level, the mother and child grids are resolved sequentially with the same parallelization option. It is thus recommended to work with grids of approximately the same number of cells to facilitate the cores distribution. In the case of multiple child grids at the same nesting level, the user can choose between two possibilities for the time integration of the child grids, i.e., sequentially or simultaneously. The first solution consists of a sequential integration of each child grid by the set of cores. The second solution involves a distribution of the MPI cores among all child grids at the beginning of the simulation. The cores can be assigned between all child grids or within multiple groups of child grids. The latter option is favorable when the child grid sizes differ or when the child grids are much smaller than the mother grid.

To evaluate the performance of this implementation, several tests were performed with the regional and coastal configurations, presented in the following sections, with classical offline one-way nesting and two-way nesting. First, the coarser model was run independently over the whole computation period to obtain information to force the child grid area at an hourly frequency. Second, baroclinic currents, temperature and salinity variables were interpolated horizontally and vertically onto the child grid with an external homemade Fortran tool. For the regional configuration usually composed of seven child grids, only the one focused on the Iroise Sea (see Fig. 5) was kept for this experiment.

Table 1 summarizes the computational times for MARS3D one-way classical offline nesting and MARS3D-AGRIF two-way nesting. For this experiment, the time integration lasts 45 d. All the simulations were performed on the Datarmor supercomputer, which is composed of 396 nodes with 28 cores each (https://www.ifremer.fr/fr/infrastructures-de-recherche/le-supercalculateur-datarmor, last access: 15 February 2023). Since the computing performance depends on the load of the machine, the computational cost was evaluated from a pool of repeated experiments (five times). For each configuration, the different models (one-way and two-way) were run with the same MPI rank distribution but from two different parallelization methods, namely the classic MPI domain decomposition and the hybrid computation (MPI and OMP) mode.

For both configurations, the computation time is increased by one-third for the AGRIF configuration compared to the classical offline one-way nesting. This difference is explained by two extra steps performed by the AGRIF library, i.e., the spatial interpolation to prepare the fine-grid boundary conditions and the update process from the fine grid to the coarse one. It is worth noticing that the handling time of the 3D output files and the waiting time due to the activity on Datarmor were not included in the one-way nesting duration. Their introduction would decrease the difference, especially if several magnifications are implemented.

As shown with the regional example, using hybrid parallelization with 8 times more computational resources than with MPI decreases the decreases the total time by 4 times. The hybrid mode is an efficient way to require more resources without considering the domain decomposition in complex geographic areas. Finally, the advection–diffusion evolution of 13 passive tracers in the coastal configuration is another example in which the hybrid mode makes it possible to achieve reasonable calculation times.

The MARS3D model has already been used to investigate the Bay of Biscay and its extension to the western English Channel (Huret et al., 2013; Lazure et al., 2009). Here, the MARS3D-AGRIF capability is implemented along the northwestern European continental shelf. Figure 5 shows how the AGRIF skill is used to pave the coastline from Spain to Belgium with overlapping grids. Seven 500 m resolution magnifications of approximately the same grid size are embedded into a coarser 2.5 km resolution grid. A total of 40 generalized σ layers discretize the vertical axis with a stretching function that induces refinement above 150 m depth next to the surface. Even though the space-refinement factor is 5, the time refinement is adapted according to the maximum velocity encountered within each grid. It is either a factor of 3 (over areas with relatively slow flows) or 5 (in very energetic areas such as the middle of the English Channel). Similarly, the turbulent viscosity coefficient (Laplacian operator) differs between each magnification and ranges from 0.5 up to 3 m2 s-1. At the surface, Météo-France atmospheric forcings drive the dynamics, with an ARPEGE (Action de Recherche Petite Echelle Grande Echelle) high-resolution (0.1∘; hourly) analysis for the coarser grid and AROME (Application of Research to Operations at Mesoscale; 2.5 km; hourly) analysis or the child grids. The main hydrological runoffs are set in the magnifications (96 rivers). They come from different source databases (Spanish, English, French and Dutch). At the open boundaries of the coarse grid, daily temperature and salinity conditions are provided by the Mercator PSY2V4 re-analysis. The tidal signal is issued from a larger 2D model (5 km resolution) forced by the FES2014 (finite element solution) ocean tide atlas with 14 harmonic constituents (Lyard et al., 2021). The sea level is imposed, while a zero gradient condition is applied to normal and tangential currents.

A realistic two-way hindcast has been realized over the period 2010–2019. To demonstrate and characterize AGRIF nesting benefits, a one-way configuration has been run on the Iroise grid over the years 2017–2018 as a reference. These 2 common years have been selected from the available datasets for several validation parameters (detailed in Appendix A). A 3-month spin-up was performed for each hindcast. The forcing and parameterization of the offline one-way hindcast are those used in the two-way configuration, apart from the tidal sea level signal, which comes from the 112 harmonic components of the SHOM CST France model (le Roy and Simon, 2003). The temperature and salinity open-boundary conditions come from the above mother grid integrated without child grids. This tidal model has been accurately validated throughout the French tidal gauge network of RONIM.

The tidal signal (extracted from the Iroise 500 m grid) is compared to hourly validated data of four tidal gauges available in the studied area. The main statistics are estimated over 1 year and given in Table 2. The tide modeled with the two-way technique is slightly less precise than with the one-way simulation. It is especially true at the northeastern boundary, at Roscoff, where the RMSE is twice as large. On the opposite side, at the southern boundary, at Concarneau, the differences are in favor of the two-way nesting.

To observe the influence of the tide propagation, the signal modeled by Iroise magnification is compared to the PREVIMER harmonic component atlas. This atlas was built upon a series of barotropic simulations at 250 m horizontal resolution and validated with the RONIM network (Pineau-Guillou, 2013). A harmonic decomposition was applied to each hourly time series extracted from the hindcasts covering an entire year. Table 3 compares the wave elevations for the waves M2, S2, O1 and K1 at four locations in the Iroise Sea. Both models are in fairly good agreement with the reference atlas. The main difference is found for the wave K1 and MARS3D-AGRIF two-way configuration. The wave K1 is over-estimated by 20 % for the relative difference, but the amplitude itself differs by less than 2 cm.

A similar comparison has also been done for the barotropic currents. The differences are higher for the minor waves of O1 and K1. They reach up to 5 cm s-1 and 30 % in relative difference at some locations (not shown). However, the validation of the PREVIMER atlas currents was not available all over the area, due to a lack of long time series data for the French coast. Barotropic currents have therefore been compared to different available Acoustic Doppler Current Profile (ADCP) datasets recorded in the bay of Brest or Molène archipelago over shallow waters. No significant difference has been found between nesting techniques.

A detailed qualification of the regional Bay of Biscay configuration has been done by Bezaud and Pineau-Guillou (2015). It highlighted the enhancements in predictions with increasing resolution in the coastal areas where the 500 m magnification models have been implemented. These comparisons have been made against coarser models. As expected, the authors concluded that a finer resolution allows the model to simulate the small-scale structures (instabilities of the front, eddies, filaments, etc.) accurately.

Here, the two-way versus one-way nesting is evaluated over 2017–2018. The evaluation focuses on the Iroise Sea magnification with different mooring stations (see the black points in Figs. 5 and 8). Figure 6 displays Taylor diagrams for temperature and salinity. A third diagram represents the bias values and root mean squared errors (RMSEs). These graphs summarize the comparisons between the available datasets (see Appendix A) and both nesting methods.

The nesting impact is not homogenous all over the domain. First, the two-way nesting technique improves the model's overall performance compared to oceanic datasets (see the green points for one-way and yellow points for two-way configurations). The simulated temperature offsets are noticeably reduced. The primary favorable impact is the drop in the root mean square error by 0.4 ∘C for the COAST-HF-MAREL-Iroise buoy (where COAST-HF is Coastal Ocean Observing System – High Frequency). For salinity comparisons, the RMSE and the bias are of the same order of magnitude. The significant improvement relies on the enhancement of correlation for the COAST-HF-ASTAN dataset. This could be due to the proximity of this point to the eastern border of the magnification. The update capability of AGRIF two-way enables more realistic incoming fluxes at a high temporal resolution. The relative standard deviation of the simulated salinity at the MAREL-Iroise buoy is also considerably reduced.

In the Iroise Sea during the summer, the Ushant front is depicted by cold water of about 14 ∘C. Over shallow depths, the tidal currents are intensified and very strong around Ushant and the Molène archipelago. The induced tidal stirring is so large that waters are mixed (and homogeneous) from the sea surface to the bottom. Further offshore, the summer stratification can develop, and the sea surface is warmer (above 18 ∘C). This phenomenon can be seen from satellite data on 15 August 2016 (Fig. 7) for both Landsat 8 and ODYSSEA products (described in Appendix A). Compared to ODYSSEA, the Landsat surface temperature is overestimated in different spots near the coast, in the bay of Brest, for example, with values over 20 ∘C which might be due to mis-flagged clouds.

The ability of the two-way nesting approach (Fig. 7d) to correctly reproduce this spatial feature is evident, while the one-way nesting (Fig. 7c) struggles to simulate this phenomenon. Indeed, the Ushant front is nearly missing in the one-way simulation. It is much better characterized in the two-way simulation at 5.25∘ W, with more realistic temperature magnitudes on each side of the front, thanks to the AGRIF update. Furthermore, the realism of the sub-mesoscale structures around the shoals of Sein Island (Île de Sein) and in the Molène archipelago is improved. On the other hand, one can notice the open-boundary effect in the southern part of the one-way simulation, where an east–west temperature front is created. It is related to the upwind scheme applied to temperature and salinity open-boundary conditions (i.e., applying an external value when the water mass enters the domain).

For wide-open areas such as the Iroise Sea, the benefit of the feedback or update of the magnification not only contributes to the correct representation of the thermic Ushant front, mainly by enlarging the span of the scale captured (e.g., fine eddies and filaments), which is expected, but also in terms of the horizontal localization of thermal front, which is less expected. As a matter of fact, the forcing through the open boundaries is of primary importance for limited area models; it may dramatically impact the local coastal processes, even in areas where the dynamics are highly controlled by large scales such as the tidal forcing. Even though the tidal signal is better represented with the one-way nesting (due to the CST France model), the upwind scheme applied for tracers at open boundaries is not accurate enough for hourly heat exchanges. The chosen simulated domain is not large enough for one-way nesting to reproduce a sharp temperature front.

The bay of Brest is a semi-enclosed, macro-tidal ecosystem located at the western end of Brittany (France), covering more than 180 km2. It is connected to the Iroise Sea through a 1.8 km wide by 6 km long and roughly 50 m deep inlet (called the Goulet de Brest). Due to its complex geometry and topography, the currents are strong, but relatively well known, and have been the focus of many previous studies. A dominant semi-diurnal tide characterizes this area, with a tidal range of 1.2 to 7.3 m. The tidal currents peak up to 3 m s-1 in the Goulet and are in quadrature phase relative to the surface elevation (Petton et al., 2020). The mean volume at mid-tide inside the bay is roughly 2×107 m3. As its average depth is only 8 m, the back-and-forth flow for each tide prevents stratification nearly everywhere (Le Pape and Menesguen, 1997). The tidal prism is 25 % of the mean volume in neap tide and 60 % in spring tide. Freshwater runoffs, mainly coming from the Aulne river, modify the hydrology locally (Auffret, 1983).

The MARS3D-AGRIF model has been set up over the Iroise Sea (47.74–48.82∘ N; 4.08–5.55∘ W) with a horizontal grid resolution of 250 m. As shown in Fig. 8, a magnification over the bay of Brest (48.20–48.44∘ N; 4.09–4.72∘ W) is introduced, with a resolution of 50 m. The time and space refinement factors are both equal to 5. The vertical discretization is performed with 20 equidistant σ layers in both grids. The bathymetries have been interpolated from a combination of digital terrain models (SHOM, Ifremer, and IGN, the French National Institute of Geography and Forestry). This Iroise model is forced by harmonic components from the SHOM CST France atlas (le Roy and Simon, 2003). The 3D open-boundary conditions for baroclinic currents, temperature and salinity are imposed at an hourly frequency from a hindcast of the previous regional configuration (Caillaud et al., 2016). Freshwater inputs for the four main rivers in the bay have been taken from the French HYDRO database (http://www.hydro.eaufrance.fr/, last access: 15 February 2023) and corrected with corresponding watershed surface rates. The atmospheric forcings rely on the Météo-France AROME (2.5 km; hourly) analysis. The configuration is available from Petton and Dumas (2022a), along with a detailed description of the physical and numerical parameterizations.

Two realistic hindcasts have been run from 2017 to 2019 for both one-way and two-way nesting techniques. Previous studies have already validated the coastal bay of Brest one-way configuration in detail (Petton et al., 2020; Frère et al., 2017). In fact, the large-scale features of the macro-tidal flow are easily captured (Petton et al., 2020) in such a semi-enclosed area. And, due to high turbulent mixing in the area, the dynamics are less sensitive to boundary effects. Consequently, the two-way nesting technique barely enhances the results. The tidal features are the same (not shown), while the hydrology is slightly modified (blue and red points for one-way and two-way nesting, respectively, in Fig. 6). Indeed, the two-way nesting reduces the bias in temperature. The feedback, which enables a more accurate global temperature budget in the mother grid, could explain this improvement. Moreover, the two-way technique reduces the relative standard deviation in salinity for the MAREL-Iroise and ECOSCOPA datasets. As the water runoffs are identical, this amelioration might be hard to explain. Nevertheless, it may result in a better simulation of current flows, according to nonlinear effects.

As the shallow ocean is subject to considerable environmental and anthropogenic pressures, the fate of coastal waters is a key to environmental, ecological and economic issues. Therefore, global and local indicators are crucial for stakeholders to anticipate the spread of different materials such as oil (Jordi et al., 2006), micro-plastics (Frère et al., 2017), biogeochemical processes due to nutrients release (Le Pape and Menesguen, 1997; Fiandrino et al., 2017) and pollution phenomena (Jiang et al., 2017; Neal, 1966) and to develop restoration solutions (Kininmonth et al., 2010; Rossi et al., 2014; Thomas et al., 2014).

The issue addressed here is the evaluation of the capability of the MARS3D-AGRIF tool to characterize the renewal capacity of the bay and identify the role played by the tidal forcing and river runoffs. The indicators have been estimated with both nesting methods for the four scenarios. The flushing times of the whole control volume are given in Table 5. As expected, low river runoffs imply a longer renewal time than flood situations. However, the initial tidal phase is the main change factor between the scenarios, with more intensive mixing during spring tides. There is a positive offset (roughly 10 %) of the renewal time from AGRIF two-way simulations. This shift is due to the return flow within the bay during each tidal cycle, which is underestimated in the one-way model, as its boundaries act as a sink for the tracer.

The local e-folding flushing times estimated with the two-way nesting configurations are displayed in Fig. 9 for the four scenarios. The indicator increases as the geographic position moves away from the control volume boundary. In contrast to the global indicator, the runoff impact on the renewal capacity of the bay is obvious. In low river runoff conditions, the southeastern part of the bay is isolated from the rest. The local e-folding flushing time reaches more than 25 d in shallow coves and is released at neap tide. The impact of tide is the next level in order of importance, with, not surprisingly, stronger renewal during spring tide than neap tide. In each scenario, the central energetic eddy stands out because it is the rallying point of continental waters.

The relative differences in local e-folding flushing time estimates are illustrated in Fig. 10. Whatever the environmental conditions, the one-way nesting overestimates the renewal capacity of the bay by predicting lower local e-folding flushing times nearly everywhere. Such bias is expected because the one-way nesting does not correctly follow the tidal prism in the control volume: any tracers that leave the volume are lost for the estimate of the e-folding flushing time. In the inner part of the bay, the local differences are always negative and reach around 20 % in flood conditions. Under low river runoff conditions, the renewal time is independent of the tide regime in the central part. In this area, the water mass is always replaced by the inflow during the period of simulation, so the treatment at the open boundary does not matter. In contrast, e-folding flushing time estimates are underestimated in shallow areas, especially during spring tides. To ensure that the disparities are due to open-boundary treatment, a similar experiment was run with a much smaller control volume located only within the bay. Then, the simulated domain is larger than the control volume, and the results are identical, regardless of the nesting method. This highlights the asset of the two-nesting approach for the evaluation of the water renewal times. Last, some suspicious patches appear at the western part of the control volume, which only takes into account tidal inputs. Others are visible next to the river mouths. Both are related to the initial time release and inaccurate exponential regression, as the inflow quickly replaces the initial water masses (as confirmed by de Brauwere et al., 2011).