This article describes the implementation of grid refinement in the ICOsahedral Nonhydrostatic (ICON) modeling system.
It basically follows the classical two-way nesting approach known from widely used mesoscale models like MM5
or WRF, but it differs in the way feedback from fine grids to coarser grids is applied.
Moreover, the ICON implementation supports vertical nesting in the sense that the upper boundary of a nested domain
may be lower than that of its parent domain. Compared to the well-established implementations on quadrilateral grids,
new methods had to be developed for interpolating the lateral boundary conditions from the parent domain to the
child domain(s). These are based on radial basis functions (RBFs) and partly apply direct reconstruction of the
prognostic variables at the required grid points, whereas gradient-based extrapolation from parent to child grid
points is used in other cases.
The runtime flow control is written such that limited-area domains can be processed identically to nested domains
except for the lateral boundary data supply. To demonstrate the functionality and quality of the grid nesting in ICON,
idealized tests based on the Jablonowski–Williamson test case

The ICON (ICOsahedral Nonhydrostatic) modeling framework is jointly developed by
the German Weather Service (DWD), the Max Planck Institute for Meteorology (MPI-M),
the German Climate Computing Center (DKRZ) and the Karlsruhe Institute for Technology (KIT),
targeting a unified global numerical weather prediction (NWP) and climate modeling system (GCM).
The development work started in 2004 with basic research on the model grid and the numerical
formulation of the dynamical core in a highly idealized shallow-water framework

Despite impressive advances in computational power over the last decades,
the application of global models with uniform, convection-permitting resolution on weather
or even climate timescales is still too costly to be performed on a regular basis.
To date, high-resolution limited-area models (LAMs) serve as a cost-effective alternative for exploring
the mesoscale and microscale, and they will continue to serve as a working horse for both the numerical weather prediction (NWP) and
climate community. Limited-area models have proven successful, but they are known to
have conceptional deficiencies, such as the potential ill-posedness of lateral boundary conditions

The oldest approach dating back more than

More recently, global models using locally refined unstructured meshes have been developed.
Unlike the grid stretching approach, they allow new grid cells to
be added in regions of interest (static h-refinement). This approach is pursued,
for example, by the Model for Prediction Across Scales (MPAS)

The approach we are pursuing closely resembles traditional two-way nesting, as known from
many regional mesoscale models such as MM5

The purpose of this article is to describe the implementation of grid nesting in ICON.
Compared to previous two-way nesting approaches, it is the first implementation on triangular
grids and differs in the way feedback from fine grids to coarser grids is realized.
In addition, the ICON implementation allows for vertical nesting in the sense that the upper boundary
of a nested domain may be lower than that of the parent domain, which is not supported by most previous
nesting implementations in other models. Without specific discussion, we note that a limited-area mode
is available in ICON as a by-product of the grid nesting implementation,
differing from nesting only in the way the lateral boundary conditions are provided.
A detailed description the grid nesting implementation will be provided in
Sect.

The understanding of ICON's nesting implementation requires some basic knowledge of ICON's
mathematical–physical design. We therefore start by summarizing key elements of
the dynamical core, time stepping and physics–dynamics coupling scheme.
ICON's dynamical core solves the fully compressible, nonhydrostatic Euler equations on the sphere
using either the shallow or deep
atmosphere formulation

To optimize computational efficiency, different integration time steps are used for the dynamical core on the one hand and
additional sub-grid physical processes (and tracer transport) on the other hand. The time steps will
be denoted by

The physics–dynamics coupling scheme in ICON further distinguishes between

Basic structure of a nested (or limited-area) domain, exemplified by
a section of domain 2 shown in the upper left.
Orange: boundary interpolation zone, having a fixed width of four cell rows.
Ocher: nudging zone with adjustable width that is
only active for one-way nesting and in limited-area mode.
Blue and light blue: child-to-parent feedback zone.
Light blue: nest overlap region, i.e., a region for which a higher-resolution child domain exists (see domain 3 in the schematic in the upper left).
Prognostic computations are restricted to the feedback
and nudging zone. Integers indicate the internal indexing of domain 2, which is used
to assign cells and edges to individual zones. More details on the indexing
and the indicated sorting of cell rows are given in
Appendices

The static mesh refinement in ICON is accomplished using multiple individual grids, whereby one or more higher-resolution (child) domains are overlaid on a coarser base (parent) domain. The base domain can be a regional or global domain. Model integration on the child domain is performed in addition to that on the underlying part of the parent domain; i.e., there is no “hole” in the parent domain where the child domain is located.

Each child domain has a defined parent domain providing lateral boundary conditions, but a parent domain can have several child domains. The child domains can be located in different geographical regions and can also serve as parent domains for further subdomains, but domains having the same parent are not allowed to share the same parent grid cells because this would lead to ambiguities in combination with two-way nesting. Nested domains may also be switched on or off during runtime. Conceptually, the number of nested domains is arbitrary and controlled by the grid files provided as input, but of course not all choices would make sense from a physical point of view. Each domain can be regarded as separate instances of the same model that are coupled to each other using the same numerical operators and filters, time integration scheme, and physics–dynamics coupling. If desired, however, different physical settings can be chosen individually for each domain. For example, radiation can be called more frequently on subdomains, or the convection scheme can be tuned differently or even switched off completely.

The refinement ratio between the parent domain and a child domain is
fixed to a value of

The parent–child coupling can be either one-way or two-way. A mixture of one-way and two-way coupled domains is also possible. Two-way versus one-way coupling means that the prognostic variables on the child domain are transferred back to the coarser parent domain at regular time intervals using a dedicated feedback mechanism. As a result, the solution on the parent domain benefits from the higher resolution of the child domain. In the case of one-way nesting, the feedback is switched off.

To perform the coupling, we conceptually split any nested domain into three zones, which we call the

The boundary interpolation zone is non-prognostic and is meant to store the
boundary conditions that are necessary to solve the governing equations in the child domain. Boundary conditions
are needed for the prognostic variables

The nudging zone, which is active in the case of one-way nesting only, serves to damp
differences between the driving solution in the adjacent boundary interpolation zone and the prognostic
solution in the child domain. Essentially, the prognostic model state of the child domain is relaxed (nudged)
to the parent state following the traditional Newtonian relaxation approach described by

In the feedback zone, the model state on the parent domain is relaxed towards
the updated model state on the child domain at every fast physics time step

The boundary update mechanism provides the child domain with up-to-date
lateral boundary conditions for the prognostic variables

In general, the boundary update works as follows.
Let

Concerning the interpolation operator

To prevent excessive overshoots and undershoots of

Regarding the interpolation of edge-based variables (i.e., the edge-normal
vector components

Horizontal reconstruction stencil for edge-normal vector components at

Edge-normal vector components at the inner child edges are
reconstructed using a direct RBF reconstruction based upon the
five-point stencil indicated by solid red dots in
Fig.

For the outer child edges, a more sophisticated reconstruction is applied
in order to ensure that the mass flux across a parent edge equals
the sum of the mass fluxes across the corresponding child edges.
We start with an RBF reconstruction of the

The edge-normal vector component

Rather than interpolating

For the horizontal mass flux

If two-way nesting is activated,
the model state

Let

For edge-based normal velocity

For cell-based variables the upscaling consists of a modified barycentric
interpolation from the four child cells to the corresponding parent cell:

Of course, the contribution of the point

In summary, this method can be regarded as a

We note that the cell-based operator

Another difficulty that was encountered in the context of mass conservation
is related to the fact that the density decreases roughly exponentially
with height. In the presence of orography, the atmospheric mass resolved
on the model grid therefore increases with decreasing mesh size,
assuming the usual area-weighted aggregation of the orographic raw data
to the model grid. Feeding back

When combining the abovementioned steps, the feedback mechanism for

Finally note that the upscaled density
includes the correction term

Care is required in order to achieve consistency with continuity.
For this purpose, feedback is not implemented for the tracer mass fractions directly,
but for partial densities. Building upon the implementation for

A very similar approach is used for

The same approach is taken for

In the case of

If the feedback is turned off, i.e., if one-way nesting is chosen, a nudging of the
prognostic child grid variables towards the corresponding parent grid values is
needed near the lateral nest boundaries in order to accommodate possible
inconsistencies between the two grids, particularly near the outflow boundary.
Nudging is performed every fast physics time step of the child domain

To compute the nudging increment, the child grid variables are first
upscaled to the parent grid in the same way as for the feedback
(Eqs.

The vertical nesting option allows setting model top heights individually for each domain, with the constraints that the child domain height is lower than or at most equal to the parent domain height and that the child domain extends into heights at which the coordinate surfaces are flat. This allows, for instance, a global domain extending into the mesosphere to be combined with a child domain that extends only up to the lower stratosphere, which can save significant computational resources. However, a vertical refinement in the sense that the vertical resolution in the child domain may differ from that in the parent domain is not implemented.

Vertical nesting requires appropriate boundary conditions for all prognostic variables
to be specified at the vertical nest interface level, i.e., the uppermost half level of the nested
domain. This is crucial in order to prevent vertically propagating sound and gravity waves from
being spuriously reflected at the nest interface. In the following, boundary conditions are derived for

Due to the constraints mentioned above, boundary conditions can be derived by
horizontal parent-to-child interpolation, without the need for any boundary
interpolation zone extending vertically away from the upper nest boundary.
For

A slightly different approach is taken for

For the tracer variables we refrain from interpolating the partial mass fluxes

The previous sections have focused on the coupling of a single child domain.
The nesting capability of ICON, however, is not limited to a single domain but
supports multiple nests at the same level and multi-level nesting, as well as a combination
of both. In the literature, multi-level nesting is also referred to as telescoping nesting

The coupling of multiple same-level nested domains with a parent domain is rather straightforward,
as it only requires the single-nest coupling strategy (Sect.

Figure

The whole processing sequence for integrating all domains from time step

Schematic of a multi-domain multi-level setup with two domains nested successively
into a global (or limited-area) base domain. The processing sequence for the time
integration of all domains from time step

The flow control of ICON's hierarchical nesting scheme is handled by a recursive subroutine that
cascades from the global domain (or outer limited-area domain) down to the deepest nesting level and
calls the dynamical core and the physics parameterizations for each domain in basically the
same way as for the global domain.
The basic processing sequence is as follows.

A single integration step with

Boundary data are interpolated from the global domain to nest

As another nested domain exists within nest

Feedback is conducted from nest

This is followed by a second lateral boundary data interpolation
from nest

As a final step, feedback is performed from nest

We note that the presented coupling strategy is very similar to that in WRF or FV3,
but differences exist in several details. For example, the child-to-parent feedback in FV3
covers only temperature and the wind components

To demonstrate the functionality of the grid nesting and to investigate the numerical errors related to the mesh size discontinuity
along the nest boundary, we start with the

List of model configurations for the Jablonowski–Williamson test.

Surface pressure (hPa, left column) and relative vorticity at

Our first series of experiments considers the baroclinic wave test for a variety of configurations based on the model grids R2B4 (mesh size 160

Relative vorticity at

As discussed in

The relative vorticity fields (right panels of Fig.

To further examine the flow disturbances generated by the resolution jump at the nest boundary, Fig.

Surface pressure (hPa) after 9 d of integration for the steady-state JW test and experiments E1

In order to examine the behavior of the upper boundary condition for vertically nested domains,
we conducted the Schär mountain test case

Following

We have performed two types of simulations: one with and one without a vertically nested domain. The reference
simulation without a vertical nest was performed on a single

For the nested simulation, a

In general, the results for the nested domain and the reference result are fairly close to each other. There is no
indication of substantial wave energy reflection or noise accumulation along the nest interface level.
Deviations from the reference result are largely confined to the uppermost quasi-hydrostatic wave crest and trough
and to the leeward propagating wave signal. This holds not only for the steady-state
solution, but also for the spin-up phase (not shown). The reason for the deviations is twofold:
firstly, the computation of the boundary conditions at the nest interface level inevitably
goes along with spatiotemporal interpolation errors. Second, and more importantly, the solutions on the parent
and child domains are slightly different, implying that the vertical interface condition derived from the parent
domain cannot exactly match the solution on the child domain.
These differences primarily originate from the
differences in the mesh size, but they additionally depend on the feedback timescale. Reducing the
feedback timescale strengthens the domain coupling and reduces the parent–child differences, which in turn improves the
vertical nest interface conditions. This can be seen by comparing Fig.

From this test it can be concluded that, even without any interpolation error, the boundary conditions at the nest interface will never match perfectly due to the resolution-induced differences between the parent and child model states. On the other hand, this test has shown a small but noticeable positive impact of the child-to-parent feedback mechanism on the quality of the nest interface conditions, which improve with decreasing feedback timescale.

Vertical velocity

Vertical velocity

In an operational context, one ideally expects the beneficial impact on forecast quality of the regionally refined resolution
to be transferred to the global domain in the nest overlap area and subsequently propagate downstream, which is usually eastward in the
extratropics. This implicitly assumes that the scores used to quantify forecast quality do improve with increasing model resolution,
which, according to our experience, is the case for mesh sizes coarser than about

The current operational configuration at DWD uses 90 vertical levels with a model top at

To exemplarily demonstrate the resolution dependence of the forecast quality and the related nest impact, Fig.

Verification against IFS analyses for the Northern Hemisphere (latitude

Verification against IFS analyses for Europe (35–72

More detailed information on the nest impact is provided by the SYNOP and TEMP verification,
which includes an assessment of statistical significance at the 95 % level. The SYNOP verification (Fig.

Scorecard for verification against SYNOP observations for

Station-wise verification of the surface pressure RMSE difference (Pa) between experiments R2B6N7 and R2B6 for forecast lead times (LT) of 24, 48, 72 and 120 h. Blue (red) dots indicate smaller (larger) errors for R2B6N7 than R2B6. Dashed boxes show the location of the nested domain.

The TEMP verification (Fig.

Scorecard for verification against radiosonde ascents for wind direction (DD), wind speed (FF),
relative humidity (RH), temperature (

Further analysis has been undertaken to investigate potential numerical issues related to the nesting, such as possible artifacts along the
lateral boundaries of the nested domain and the loss of exact mass conservation mentioned in Sect.

Accumulated total precipitation (kg m

To examine the impact of the two-way nesting on mass conservation, an additional set of forecast experiments has been conducted in which the lead time
was extended to 30 d and the nested domain stays active until the end. Results for 18 arbitrarily selected initial dates are summarized in
Fig.

Relative mass conservation error in the global domain for 18 selected 30 d forecast experiments in January 2021 in which the nested domain remained active throughout the forecast.

This article provides a technical description of the grid nesting implementation in the ICOsahedral Nonhydrostatic (ICON) modeling system.
The available options comprise one-way and two-way nesting, with one or more domains per nesting level, and vertical nesting in the sense
that the upper boundary of a nested domain may be lower than that of its parent domain. In addition, a limited-area mode is available
as a by-product of the grid nesting implementation, which differs from one-way nesting only in the way the lateral boundary conditions
are provided. The model-internal flow control basically follows
the nesting approach known from mesoscale models like MM5

To demonstrate the functionality and quality of the grid nesting in ICON, idealized tests restricted to the dry dynamical core are presented as are
real forecast experiments using a model configuration very close to that used for operational NWP at DWD. The horizontal grid nesting is addressed by
two variants of the Jablonowski–Williamson test case

In order to identify grid points belonging to certain regions of a domain and to
control the feedback between parent and child domains, dedicated integer fields named

In regions with an overlapping child domain the counters for cells, edges and vertices are filled with
negative numbers (see light-blue area in Fig.

Several measures are taken in order to optimize the computational efficiency of the nesting implementation.

Model grid points in ICON are stored in (blocked) 1D vectors. In order to achieve
efficient runtime flow control without the need for masking operations within loops,
the grid points are ordered by their distance from the lateral boundary by making use of
the

Regarding distributed-memory (MPI) parallelization, the default strategy adopted in ICON is to distribute all model domains among all compute processors. As this implies that child grid points are in general owned by a different processor than the corresponding parent grid point, an intermediate layer having the mesh size of the parent grid but the domain decomposition of the child grid is inserted in order to accommodate the data exchange required for boundary interpolation and feedback.

To reduce the amount of MPI communication for complex nested configurations, multiple nested domains at the same nesting level can be merged into one logical domain, which is then not geometrically contiguous. This needs to be specified by the user during the grid generation process by indicating a list of individual domains that are supposed to be merged. The lateral boundary points belonging to all components of the merged domain are then collected at the beginning of the index vector. For all prognostic calculations, the multiple domains are treated as a single logical entity, and just the output files may be split according to the geometrically contiguous basic domains. As one-way and two-way nesting cannot be mixed within one logical domain, there may still need to be two logical domains on a given nest level.

To further optimize the amount of MPI communication, a so-called processor splitting is available that allows for executing several nested domains concurrently on processor subsets whose size can be determined by the user in order to minimize the ensuing load imbalance. Unlike domain merging, this allows parallel execution of one-way and two-way nested domains. This option is currently restricted to the step from the global domain to the first nesting level in order to keep the technical complexity at a manageable level.

The ICON release version icon-2.6.4 is freely available under a personal noncommercial research license.
Information on the license and instructions for downloading the code can be found at

All authors contributed to the writing of this paper. GZ conducted the initial nesting implementation
and ran the experiments of Sect.

The contact author has declared that none of the authors has any competing interests.

Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The authors would like to thank the entire ICON team for their excellent work, without which this would not have been possible. Our additional thanks go to Felix Fundel (DWD), who developed the verification toolchain used in this work. The authors also appreciate the constructive comments made by the two anonymous reviewers, which led to a significant improvement of the paper.

This paper was edited by Ludovic Räss and reviewed by two anonymous referees.