The implementation of a “classical” Discontinuous Galerkin (DG) solver for atmospheric flows is presented that is designed for efficient use in numerical weather prediction, climate simulations, and meteorological research both on the whole sphere and for limited area modeling. To this purpose the horizontally explicit, vertically implicit (HEVI) approach is used together with implicit-explicit (IMEX)-Runge-Kutta (RK) time integration schemes and a moderate spatial approximation order (order 4 or 5). Two Euler equation sets using mass, momentum and either density weighted potential temperature θ or total energy E as prognostic variables are compared by several idealised test cases. Details of the formulation of the Euler equations in covariant form using the Ricci tensor calculus, the linearisations needed for HEVI (especially for the total energy set), boundary conditions for an IMEX-RK scheme, and filtering for numerical stabilisation are given. Furthermore, the implementation of distributed memory parallelisation, the tensor product representation for prismatic grid cells, and optimisations for the HEVI formulation, are outlined. These developments lead to the so-called BRIDGE code, which will serve as a code base for a later DG extension of the well established ICON model. From the used idealised test cases, which are standard benchmarks for dynamical core development for the atmosphere, we conclude that the equation set using total energy E has better well-balancing properties than the set using θ. This result can be confirmed by a normal mode stability analysis. However, in some tests the set using E suffers more from non-linear instabilities that can only be partially solved by filtering.
Bundesministerium für Forschung und Technologie01LK2315CIntroduction
Among the numerous different methods to numerically solve partial differential equations, the Discontinuous Galerkin (DG) method has attracted a certain attention in many disciplines (fluid mechanics, elastomechanics, acoustics, electrodynamics, …) during the last decades since its first invention at the beginning of the 1970's (for a historical overview see ). The DG method allows to use quite arbitrary grid cells (triangles, prisms, hexagons, …) on arbitrary unstructured grids and runs efficiently on massively parallel computers due to very compact stencils. From a more numerical point of view DG methods can easily achieve higher order approximations, which highly increases accuracy for well resolved fields. However, in particular in fluid flows the fields are sometimes (or often) underresolved – here, another property of the DG method, namely its ability to treat every prognostic variable in a locally conserving manner (or better say: the flux divergence terms do not violate local conservation) helps in improving the physical consistency. Furthermore, DG methods in principle allow explicit time integration schemes, therefore one has a certain flexibility in the choice of an appropriate (explicit or implicit or combined) time discretisation.
After a few preliminary studies at the Deutscher Wetterdienst (DWD) about the usability of the DG method for atmospheric models it was decided to start a new model development at DWD, called BRIDGE (Basic Research for ICON with DG Extension) . The new implementation is intended to be part of a DG version of the established ICON model later on. Therefore, its design aims to closely resemble and seamlessly integrate with ICON's Fortran codebase. The starting point for BRIDGE was the application of a DG scheme to solve the shallow water equations on a triangular grid on the sphere by (in the following denoted as B20), and the solution of the Euler equations in terrain-following coordinates via the horizontally explicit, vertically implicit (HEVI) approach in a DG solver using implicit-explicit (IMEX)-Runge-Kutta (RK) time integration schemes by (in the following denoted as B21). One result of these preliminary studies was that a spatial approximation order of about 4 seems to be adequate; grid irregularities don't play a role any more, but the cells are still small enough to benefit from local conservation. Additionally, the results showed that grid staggering is not necessary any more, a fact that simplifies coupling of parameterizations and coupling with geometrically separated subsystems like land, lake or ocean models.
One question in the design of a new dynamical core for atmospheric models is the choice of the prognostic variables, here in particular for the non-hydrostatic, compressible Euler equations (also see the different versions proposed in or ). Since a DG scheme in general allows a better separation between the equation formulation and the pure numerical implementation this question seems less serious and is less a basic design issue than for other discretisation schemes (as e.g. finite difference schemes). Of course, one wants to keep the local conservation properties of the finite volume part of a DG scheme, therefore a flux form of the equations using density like variables (mass density, momentum instead of velocity, …) is preferred. However, it turns out, at least for the “classical” DG formulation used in the following, that the choice in particular of the thermodynamic variable is less arbitrary than expected. In B21, the density weighted potential temperature ρΘ was used, a choice that is made by several atmospheric simulation models as ICON . Here, the use of ρΘ will be compared with the use of total energy density E (i.e. the sum of kinetic, potential and internal energy contributions), the latter is the preferred variable in computational fluid dynamics (CFD) simulation models.
The motivation to use total energy E of course stems from the fact that energy is one of the most fundamentally conserved variables in physics. Its conservation can be derived from time invariance of the physical laws (Noether's theorem), formulated as the first law in thermodynamics. Total energy further has the advantage to be quite unambiguously defined (there is no doubt about how to calculate kinetic, potential, internal, latent, … energy parts). On the other hand, if one restricts oneself to reversible processes and if the air does not change its constituents, the use of ρΘ implies local conservation of entropy (since in this case specific entropy s is related to potential temperature via s=cplogΘ, where cp is the specific heat capacity at constant pressure). Interestingly, local conservation of ρΘ still holds in the presence of diffusion (see Eqs. , , below), although entropy is not conserved for an irreversible process. However, in the more interesting atmospheric processes with change of moisture constituents, ρΘ is no longer an exactly conserved quantity. Beyond this, the definition of Θ is no longer unique in a moist atmosphere due to several possibilities which thermodynamic coefficients (R, cp) should be used (the dry values or the moist values? Existence of other chemical or aerosol components, …). Although this is not a fundamental problem, it can become a practical one, if several parameterisations are coupled with a dynamical core (see also for a few more properties of these two variable choices).
Therefore the purpose of this article is twofold. First, a DG implementation of a solver for atmospheric flows (BRIDGE) is presented that is designed for efficient use in numerical weather prediction, climate simulations, and meteorological research both on the whole sphere and for limited area modeling. Second, the influence of the choice of the prognostic variable ρΘ or total energy E in this framework will be investigated.
The article is organized as follows: Sect. presents the two Euler equation sets in covariant (i.e. coordinate system independent) form and explains the concrete coordinate systems used in BRIDGE. Although there exist several attempts worldwide to develop DG based atmospheric models, our approach is different to most of them by using prismatic grid cells, to later on support the ICON user community. Our consistent usage of the covariant formalism allows to formulate the Euler equations on arbitrary manifolds while keeping basic DG properties, an aspect that might be of interest for other modelling communities, too. Section explains the HEVI approach and in particular gives the needed linearisations for the energy form of the Euler equations. Our determination of the linearisation coefficients is slightly different from some other HEVI methods by omitting several terms, which can lead to a slightly higher efficiency. Several implementation details about parallelisation, usage of tensor products, and optimisations of the HEVI scheme are presented, too. The treatment of boundary conditions in IMEX time integration schemes is explained in Sect. . Filtering is used to prevent (mostly non-linear) instabilities and is explained in Sect. . Finally, Sect. presents results from several atmospheric test cases.
Analytic formulationsTwo Euler equation sets
In the following we write the Euler equations in covariant form, therefore valid in every arbitrary coordinate system. To this purpose we make extensive use of the Ricci tensor formalism (see B20, B21 and references therein), in particular we use the Einstein summation convention and sum over similar upper (contravariant) and lower (covariant) indices. Mass conservation is expressed by the continuity equation
∂ρ∂t+∇kMk=0
with mass density ρ and momentum density Mk. ∇k denotes the covariant derivative; occasionally we use the denotation with a semicolon for it, e.g. ∇kMi≡M;ki.
The momentum balance equation is expressed as follows:
∂Mi∂t+∇kTik=S(M)i
with the momentum flux tensor for inviscid flow
Tik=Mivk+pgik,
where vk=Mk/ρ denotes the velocity, p the pressure, and with the momentum source term consisting of buoyancy and Coriolis force
S(M)i=-ρgik∇kΦ-2gijEjklΩkMl.
Here, Φ is the gravitational potential, Ωk are the components of the angular velocity of the earth, gij the contravariant components of the metric tensor, Ejkl are the covariant components of the totally antisymmetric Levi-Civita tensor of rank 3 (analogous to the 2nd rank tensor given in B20).
In case of diffusion due to (turbulent) viscosity we add
T(D)ik=-ρKa2Dik-ρKbgik∇lvl
to the above momentum flux Tik with the symmetric deformation tensor
Dik=12gil∇lvk+gkl∇lvi.
As in B21, we use Ka=KM and Kb=-23KM with the kinematic shear viscosity KM. In the test cases of Sect. we either set KM=const. or use a mixing length approach
KM=lt2D,D=2DijDij
(note the typo in the definition of the scalar shear D in B21). Here, lt denotes a turbulent length scale (we specify one in Sect. ).
In the following we will consider two variants of the energy balance:
Equation set “<inline-formula><mml:math id="M45" display="inline"><mml:mrow><mml:mi mathvariant="italic">ρ</mml:mi><mml:mi mathvariant="normal">Θ</mml:mi></mml:mrow></mml:math></inline-formula>”
Equation set “ρΘ” uses the density weighted potential temperature ϑ=ρΘ as the prognostic variable (this is the approach used in B21)
∂ϑ∂t+∇kf(ϑ)k=S(ϑ),
with the ϑ-flux for the pure Euler equations
f(ϑ)k=vkϑ.
In case of diffusion we add
f(ϑ,D)k=-ρKHgkl∇lΘ.
This expression is motivated by the fact that there is no turbulent heat flux in a neutrally stratified atmosphere. In this article we use the simple (but generally not correct) assumption that the diffusion coefficient for heat KH has the same value as KM.
The pressure results from the equation of state for an ideal gas
p=prefϑRdprefcp/cv
with a reference pressure usually chosen as pref=105Pa.
At the moment we only consider adiabatic processes (more specific: we have no energy sources/sinks due to latent heat release, chemical reactions, radioactive decay or radiation absorption/emission). Therefore, it holds S(ϑ)=0.
Equation set “<inline-formula><mml:math id="M58" display="inline"><mml:mi>E</mml:mi></mml:math></inline-formula>”
As an alternative to Eq. () we use the balance equation for the total energy density E∂E∂t+∇kf(E)k=S(E).
For adiabatic processes, we have no energy sources: S(E)=0.
The total energy is the sum of kinetic, potential and internal energy E=Ekin+Epot+Eint with components
Ekin=121ρgjkMjMk,Epot=ρΦ(r),Eint=cvρT.
For given prognostic variables we can first determine the internal energy Eint=E-Ekin-Epot and then calculate the temperature T. The pressure follows from the ideal gas equation
p=RρT
with the individual gas constant R for air. Pressure also has a simple relationship with the internal energy
p=RcvEint
and can be expressed by the prognostic variables via
p=RcvE-121ρgjkMjMk-ρΦ.
The energy flux for the Euler equations is expressed as
f(E)k=vk(E+p).
In case of viscosity and heat conduction we add two diffusive flux contributions
12f(E,Dk)k=vjgjlT(D)kl,13f(E,Dh)k=-KHρcpTΘgkj∇jΘ.
The contribution f(E,Dk)k can be derived from the full momentum flux tensor Tik+T(D)ik when considering the prognostic equation for the kinetic energy (see e.g. ). Analogously, the contribution f(E,Dh)k can be derived from the prognostic equation of the internal energy Eint using Eq. (), note that this form exactly agrees with those given in . In both these derivations, there occur source terms that contribute to changes of internal energy. However, the total energy is conserved. Written in the above given variables we can reformulate
f(E,Dh)k=-KHgkj∇jEint-cpcvEintρ∇jρ.
Use of a reference state
As in B21 we subtract a stationary, resting, and horizontally homogeneous reference state (denoted with subscript 0) that is in hydrostatic balance, and write the above prognostic equations for the deviations (denoted with a tilde), e.g. ρ̃=ρ-ρ0. In general, this improves the well-balancing problem in numerical discretisations of the pressure gradient and the buoyancy term in the momentum balance. Note that this reference state is not used for the linearisation later on in the HEVI scheme. In particular, for equation set “E” we subtract the reference state energy
E0=ρ0Φ(r)+cvρ0T0=ρ0Φ(r)+cvRp0.
One possible choice is the isothermal atmosphere
T0(z)=T00=const.,p0(z)=pref(z=0)e-gRT00z.
with the gravitational acceleration g=∂Φ/∂z and ρ0 following from the ideal gas Eq. (). However, if not otherwise noted, we use the reference state
T0(z)=Tref,0+ΔTrefe-zHref-1
from , with Tref,0=300K, ΔTref=75K, Href=10000m and the pressure at bottom pref(z=0)=105Pa.
Coordinate systems
The above equations only use tensors (or products of two pseudo-tensors) and therefore are valid in every arbitrary coordinate system. As noted in B21, it is advantageous to rewrite these equations in the so-called strong-conservation form, because this reduces numerical violation of local conservation due to metric correction terms. This means that we now have two distinct coordinate systems K and K′, where K′ is “smoother” than K. For our purposes, K is a terrain-following coordinate system, whereas K′ just follows the sphere (or even a flat plane). By this we can identically reformulate the scalar Eqs. (), (), or () as
∂gϕ∂t+∂gf(ϕ)k∂xk=gS(ϕ),
where ϕ is ρ, ϑ or E, respectively, f(ϕ)k is the related flux component, and S(ϕ) is the related source term, and the momentum Eq. () (or any other prognostic equation for a vector field) as
∂gMi′∂t+∂∂xkgTi′k=gS(M)i′-Γl′k′i′Tl′k′,
with the scalar metric density g=detgij and the Christoffel symbol of second kind Γl′k′i′.
Apart from these two types of prognostic equations we also have to deal with derivative variables for the treatment of diffusion via the BR1 scheme (, also see ). For these we use the form given in B21, Sect. 2.4, i.e. we consider the covariant derivative ϕ;j of a scalar ϕ or the covariant derivative M;jl of a vector field (with terrain-following components) Ml where the derivation is done by the terrain-following coordinate xj.
Practically, we use the following sequence of coordinate systems and the related transformations
We use different numbers of primes to indicate which coordinate system is used for the coordinates and tensor components.
:
For all input and output purposes, we use spherical coordinatesxi′′=(λ,ϕ,z),z=r-rsphereon a sphere with given radius rsphere. The metric properties of these coordinates and their transformations from/to Cartesian coordinates x=x1′′′, y=x2′′′, z=x3′′′ are given in B20, Appendix A.
Alternatively, for idealized tests we can use flat Cartesian coordinates. For convenience we also denote them asxi′′=(λ,ϕ,z),with the simple transformations from/to Cartesian coordinatesx=Rxλ,y=Ryϕ,z=z,Rx,Ry=const.
On the sphere a triangulation is performed, i.e. all horizontal (2-dim.) grid cells are triangles. We map each of these spherical triangles onto unit triangles by a local gnomonial projection and call the related coordinate the local or unitcoordinate
The technical distinction between local and unit coordinates is justified as follows: In BRIDGE the unit coordinate can be a horizontally affine transformation of the local coordinate. In the current implementation this affine transformation is just the identity.
xi′ (note that there is no change in the vertical coordinate, i.e. x3′=x3′′). Consequently every triangle uses its own mapping and has its own coordinate system and related base vectors (and is therefore an example of a differentiable manifold). This means that tensor components change from one triangle to another. The gnomonial projection has the important property that points (in particular quadrature points) of both common edges of neighbouring spherical triangles are mapped to the same distances along the unit triangle edges.
The metric properties of these coordinates and transformations between xi′ and xi′′ are given in B20, Appendix C (note that a slightly different notation of primes is used there).
To consider orography, we introduce terrain-following coordinatesxi, as in B21. They are defined byx1=x1′,x2=x2′,x3=f(x1′,x2′,x3′).The (user-defined) stretching function f may also depend on the orography horo(x1′,x2′) and the model top height Htop.
We use the usual transformation rules for (2nd rank) tensors to transform metric properties from one coordinate system to the next one,
gij=∂xi∂xi′∂xj∂xj′gi′j′,gij=∂xi′∂xi∂xj′∂xjgi′j′,
and for the Christoffel symbols (which are not tensors)
Γjki=∂xi∂xi′∂xj′∂xj∂xk′∂xkΓj′k′i′+∂xi∂xl′∂2xl′∂xj∂xk.
Note that throughout the entire DG framework (i.e. apart from input/output) we only use the terrain-following coordinate xi as the independent coordinate (e.g. for derivatives) and the unit coordinate xi′ for the momentum components. Therefore, the above denotation of coordinates matches with the strong conservation form Eqs. () and () given above.
Finally, a remark should be made about the shallow atmosphere approximation in contrast to the deep version. In the deep case, nothing else must be done in the above described approach. However, all test cases below on the sphere use the shallow atmosphere approximation, and this approximation is still used by many global forecast models worldwide due to slight efficiency reasons. This means that in all metric properties in the above coordinate system 1. the radial coordinate x3′′+rsphere is replaced by the constant earth radius rsphere. Consequently, all partial derivatives of the metric tensor by the radial coordinate vanish: ∂3′′gi′′j′′=0. This implies that those components of the Christoffel symbol vanish for which at least one index is equal to 3: Γj′′k′′i′′=0 if i′′=3 or j′′=3 or k′′=3. These metric properties are then transformed to the subsequent coordinate systems. This procedure automatically generates the right metric properties for a shallow atmosphere (in particular the right divergence and advection terms, see e.g. ). The only thing left to do is to apply the so-called traditional approximation, i.e. to skip some Coriolis terms. This is done by setting the radial unit component of the earth rotation velocity vector only: Ω1′=Ω2′=0, Ω3′=Ωsinϕ, with the geographical latitude ϕ and the earth angular velocity Ω=7.29212×10-5rads-1. The gravitational potential in the shallow case sounds Φ(r)=gz with either the radial direction z in spherical coordinates or the vertical direction z in Cartesian coordinates, and the gravitational acceleration g=9.80665ms-2.
The HEVI-DG discretisationThe HEVI approach and related linearisations
In most meteorological model applications, the grid cells, at least near the ground, have a much smaller vertical than horizontal extent. Therefore, the time step restrictions related to the vertically expanding fast sound waves in particular are severe. We use the well-known HEVI approach to overcome this and again follow B21. To this purpose, the fluxes are split into an explicitly treated part fex that contains all horizontal and the nonlinear vertical contributions, and an implicitly treated vertical, linear part fim. The source terms are analogously split into slow, explicitly treated and fast and linear, implicitly treated parts. In its most general form these equations (independently from its tensorial rank, i.e. either the scalar Eq. or the vector Eq. ) can be written for any prognostic variable q(σ) as
∂gq(σ)∂t+∂∂xkgfex(σ)k+∂∂x3gfim(σ)3=gSex(σ)+gSim(σ).
The upper index σ enumerates the Nσ prognostic variables. It should be noted that the sequence of coordinate systems described in Sect. always maps to unit cells (unit prisms in BRIDGE), therefore, we directly can identify the scalar metric density with the Jacobian J≡g, as it is usually denoted in the finite element or DG/CG literature.
In the following sections we define the implicitly treated terms. To keep consistency, all remaining (mostly nonlinear) terms are treated explicitly (e.g. ). Therefore, we have
fex(σ)1=f(σ)1,fex(σ)2=f(σ)2,fex(σ)3=f(σ)3-fim(σ)3,Sex(σ)=S(σ)-Sim(σ),
where f(σ)j and S(σ) are the complete (nonlinear) flux and source terms, and with the vertical flux and source term linearisations of the implicit terms
fim(σ)3=∑σ′=12Nσδf(σ)3δq(σ′)q(σ′),Sim(σ)=∑σ′=12NσδS(σ)δq(σ′)q(σ′).
The linearisation coefficients δf(σ)3/δq(σ′) and δS(σ)/δq(σ′) can depend on the variables q(σ), which are generally taken from a previous time step. In these sums, q(σ) either can be the Nσ=5 prognostic variables (q(1)=ρ̃, q(2)=M1, …) or, in anticipation of possible diffusion terms, the Nσ=5vertical derivative variables (q(6)=ρ̃;3, q(7)=M;31, …). demonstrate that this linear HEVI approach (denoted as “LHEVI-PS” there) has superior efficiency properties (at similar numerical stability) compared to other HEVI schemes that treat all nonlinear terms in an implicit manner.
Linearisation for the equation set “<inline-formula><mml:math id="M172" display="inline"><mml:mrow><mml:mi mathvariant="italic">ρ</mml:mi><mml:mi mathvariant="normal">Θ</mml:mi></mml:mrow></mml:math></inline-formula>”
All the linearisations needed for equation set “ρΘ” are described in B21; see Sect. 2.5.1 therein for the pure Euler equations and Sect. 2.5.2 for the related diffusion terms. We notice that these linearisations are not done in a strict mathematical sense, but in a way that the linear state is relatively close to the actual state of the atmosphere and keeps near hydrostatic balance as much as possible.
Linearisation for the equation set “<inline-formula><mml:math id="M174" display="inline"><mml:mi>E</mml:mi></mml:math></inline-formula>”
The terms that are responsible for sound expansion are the pressure gradient in the momentum equation and the velocity divergence in the thermodynamic equation (here: the total energy equation). Additionally, an implicit treatment of the buoyancy term helps in stabilizing gravity wave expansion.
Therefore, we first have to look at the pressure deviation p̃ in the momentum equation. From Eq. () and the reference state energy, Eq. (), we get
p̃=p-p0=RcvẼ-121ρMjMj-ρ̃Φ.
Though this is an exact expression, Ẽ arises only in a linear manner, the same holds for ρ̃ in the third term. This is the main reason why there is no problem with using the potential energy: any shift in the geopotential exactly cancels in the first and third term. At a first glance, a linearisation just in these two variables ρ̃ and Ẽ (i.e. without the kinetic energy term) should be sufficient for an implicit and therefore stable treatment of the vertically expanding sound waves. However, since the full pressure gradient is important, it turns out that the kinetic energy term must be linearised, too, in the momentum derivatives of the momentum flux to get a stable behaviour.
So, for the pure Euler equations (set “E”) we express the fluxes (), () by the prognostic variables ρ̃, Mj′ and Ẽ and can derive the following linearisations just by differentiation: for the mass flux
for the momentum fluxes
and for the energy flux
We want to emphasize that these linearisation coefficients depend on the actual or at least recent state q(σ)=q0(σ)+q̃(σ) of the atmosphere and not only on the reference state q0(σ). As discussed above not every term is necessary for a stable treatment of sound and gravity waves and in fact we only use the underlined terms. In particular, we neglect linearisations by E or ρ in the energy flux and skip terms that are quadratic in the momentum variables in general. The omittance of these terms is mainly done by efficiency reasons; they also seem to have in general only a minor influence. We have found experimentally that one can even set δTi′3δM3′=0 without detrimental effects, because this omission can slightly reduce a nonlinear instability when the vertical momentum M3′ starts to oscillate. However we have seen (e.g. by the simple 1D test case in Sect. 6.1 with a horizontal base flow), that the following two terms need a special treatment, which otherwise would lead to strong overshooting of vertical velocity near the top and bottom boundaries. First, the underlined term denoted as “factor 1/2” stems from the kinetic energy (by the pressure term in the momentum flux) and, as a quadratic term in the velocity, it is taken only with a factor 1/2. Second, the term ∼Ekin in Eq. () should be omitted, too. These particular choices are motivated by the fact that the implicit momentum flux then exactly results in the pressure perturbation (use Eq. )
fim(Mi′)3=δTi′3δρρ̃+δTi′3δMk′Mk′+δTi′3δEẼ=Rcv-ρ̃Φ-12ρMk′Mk′+Ẽgi′3=p̃gi′3,
which helps to improve hydrostatic well-balancing in the implicit part. We want to remark that this is a slightly different linearisation strategy as e.g. used in or .
Apart from these flux term linearisations we only have one source term to linearise, namely the buoyancy term in the momentum equation with
δS(M)3′δρ=-g3′3′∇3′Φ.
All other implicit source term coefficients in Eq. () are set to zero.
In case of diffusion, we get the linearisations as follows. There is no diffusion term in the continuity equations. The diffusive term in the momentum equation does not depend on the thermodynamic variable, therefore we use the same as for equation set “ρΘ”. The diffusive energy flux f(E,Dh)3, Eq. (), can be linearised by the prognostic and vertical derivative variables to give
f(E,D,lin)3=δfE,D3δρρ̃+δfE,D3δMj′Mj′+δfE,D3δEẼ+δfE,D3δρ;3ρ̃;3+δfE,D3δM;3kM;3k+δfE,D3δE;3Ẽ;3,
with the linearisation coefficients
δfE,D3δρ=-KHg3j-1+cpcv1ρ3Mk′Mk′ρ;j+cpcvEρ2ρ;j-g∂x3′∂xj+1ρ2M;jkMk,δfE,D3δMj′=-KHg3j1+cpcv1ρ2ρ;jMj′-1ρM;jk∂xk′∂xkgk′j′,δfE,D3δE=KHg3jcpcv1ρρ;j,δfE,D3δρ;3=-KHg33121ρ2Mk′Mk′-Φ-cpcvEintρ,δfE,D3δM;3k=KHg331ρMk,δfE,D3δE;3=-KHg33.
Note that we skip all linearisations related to the inherently quadratic diffusive momentum flux contribution fE,Dk. Up to now, we have not seen stability problems by not treating fE,Dk implicitly.
All these linearisation coefficients are calculated (“updated”) only after every 50 timesteps, to reduce the effort of performing the expensive LU decomposition. This is supported by the fact that we only use SDIRK IMEX-RK schemes, where the diagonal RK coefficients of the implicit part are all the same. Therefore, we can use the same matrices for the implicit back-substitution, matrix-vector multiplications for all RK stages during 50 time steps. However, for the baroclinic instability test, Sect. , and the test, Sect. , the time steps are so large, that a higher frequency of every 10 timesteps is partly necessary for numerical stability.
The DG scheme
In this section, we describe the basic steps towards a “classical” DG-scheme for the Euler equations in strong conservation form. Here, we only present the basic formulation, details about the HEVI formulation can be found in B21. The 3D domain is divided into Nhc×Nvc non-overlapping prismatic cells Ωjk=Ej×Ik, where the index j=1,…,Nhc enumerates the cells in the horizontal directions. and k=1,…,Nvc enumerates in the vertical direction. This means that the BRIDGE code makes the common assumption of an unstructured columnar mesh (prismatic cells): each grid cell of the horizontal mesh corresponds to a vertical column extending radially upward. As mentioned in Sect. , we assume that the coordinate transformation already maps onto unit cells in the system K. Therefore, the vertical interval is the unit interval Ik=[-1,+1] and in the horizontal we have unit triangles Ej={(x1,x2)|-1≤x1≤1,-1≤x2≤x1}.
The weak form is achieved by multiplication of the general Eq. () with a test function v over the cell Ωjk and integration by parts
ddt∫Ωjkgq(σ)vdV=-∫∂Ωjkgvfex(σ)k,numnkda-∫∂Ωjkgvfim(σ)3,numn3da+∫Ωjkgfex(σ)k∂v∂xkdV+∫Ωjkgfim(σ)3∂v∂x3dV+∫ΩjkgSex(σ)vdV+∫ΩjkgSim(σ)vdV,
where n denotes the surface unit normal vector that is directed outwards of cell Ωjk, dV is the volume element, and da the surface element.
As mentioned earlier, the classical BR1 scheme is used to handle the second-order diffusive terms. Therefore, the original set of equations is reformulated into a system of coupled first-order equations and 3Nσ auxiliary variables representing gradients are introduced.
For the numerical flux of the variable q(σ) in direction n (which directs from “L” to “R”) we use the local Lax-Friedrichs (LF) flux (also known as Rusanov flux)
f(σ)k,numnk≡f(σ)num(qR,qL,n)=12fR(σ)j+fL(σ)jnj-λ(n)2(qR(σ)-qL(σ)).
Note that the LF-flux has the usual antisymmetry property f(σ)num(qL,qR,-n)=-f(σ)num(qR,qL,n), which guarantees local conservation on a cell-wise level. This flux is used both for the hyperbolic explicit part and for the implicit part with suitable “diffusion velocities” λex and λim, respectively . The diffusion velocity for the Euler equations at the edge with unit normal vector n is known to be the sum of the speed of sound and the absolute value of the advective velocity
λtot(n)=Gcsnd+|nivi|,csnd=cpcvpρ,G=ninjgij.
The derivation in particular of the metric factors is done analogous to those in B20 (Appendix D) for the shallow-water equations. According to , λtot is split into explicit and implicit contributions
λex(n)=|Gcsndnhor|+|nivi|,λim(n)=λtot(n)-λex(n).nhor denotes the horizontal part of n.
As mentioned in Sect. , every triangle uses its own coordinate base vectors, therefore we have to transform the horizontal, physical flux components f1 and f2 and the diffusion velocity λ at the horizontal edges of two neighbouring prisms, so that each cell can calculate its own numerical flux, as is described in B20, Sect. 3. Note that such a transformation does not interfere at all with the purely vertical implicit treatment of the HEVI approach. Furthermore, this symmetric flux transformation does not violate the local conservation property (see the proof in B20, Appendix E). In contrast, in the application of the BR1 scheme for diffusion, we exchange the derivative variables instead of the related “pseudo fluxes”. The reason for this different treatment is that the flux calculations of the Euler equations (in particular the additional diffusion terms) is relatively expensive, therefore it is efficient to calculate them only once and then transform to the neighbouring cell face. A flux has one tensorial rank more than its related prognostic variable, therefore the transformation is a bit costlier than it would be for the variable itself. Conversely in case of the derivative variables, the “pseudo flux” calculation is computationally cheap and it is therefore better to perform the flux calculation twice without the transformation.
The linear system of equations (LSE) resulting from the implicit part of the HEVI equations is given in B21, Sect. 3. This LSE defines one stage for an IMEX-RK time integration scheme. In contrast to the SSP3(4,3,3) scheme that was preferably used in B21, here we will mostly use the SSP3(3,3,2) scheme , whose implicit part is L-stable and whose explicit part is a strong stability preserving (SSP) 3rd order RK scheme, too (although we don't really benefit from the SSP property at the moment, since an Euler forward step is not stable by itself). Although its overall accuracy is only second order, we use it in BRIDGE due to higher efficiency because, firstly, this 3-stage scheme saves one implicit solve per timestep compared to the 4-stage SSP3(4,3,3) scheme, and secondly it allows a larger Courant number of about 0.12 for the linear advection equation with two velocities compared to about 0.08 for SSP3(4,3,3) (see B21, Table 1).
Remarks about the numerical implementationRemarks about parallelization
One advantage of DG methods is their high computational intensity because of their high-order polynomial approximations and the large number of degrees of freedom in each element. Additionally, the locality of DG – requiring only face-neighbour data – allows for excellent weak and strong scaling on distributed-memory systems via domain decomposition. MPI (Message Passing Interface) enables DG parallelization with ghost exchanges for faces in two different forms:
In the first variant, one process calculates the numerical flux from the double-valued surface states and communicates the fluxes back to the neighbouring process . This parallelization strategy does not offer performance advantages for BRIDGE, as the fluxes for one side are not uniform due to the mapping: Since we use local coordinates on the sphere, we get different numerical fluxes for the right and the left cell.
Therefore, the BRIDGE code implementation employs the second variant, where only the trace of the solution has to be sent to the neighbouring partition while the partition boundary fluxes are evaluated twice . Note that this is not related to the previous remark about process-local flux recalculation in Sect. . We use non-blocking MPI communication calls to overlap communication and computation.
Sum factorization techniques
As mentioned in Sect. , the BRIDGE code makes use of a two-dimensional (2D) spatial mesh and a (transformed) vertical axis. This approach is common in geophysical PDE simulations, as it avoids the high computational cost of full 3D. Consequently, the discrete finite element space of the BRIDGE code is a product space built from horizontal finite elements with polynomial expansion Πm(σ)(x1,x2) and vertical finite elements with polynomial expansion Pn(σ)(x3) (potentially differently chosen for each variable q(σ)). In each cell the solution is represented as a linear combination of an Mh×Mv-tensor product base by Nh(σ) horizontal base functions Πm(σ)(x1,x2) and Nv(σ) vertical base functions Pn(σ)(x3) with Mh=∑σ=1NσNh(σ), Mv=∑σ=1NσNv(σ)qjk(σ)(r,t)=∑m=1Nh(σ)∑n=1Nv(σ)Qjk,mn(σ)(t)Πm(σ)(x1,x2)Pn(σ)(x3),r=rjk(xi).
For the sake of readability, we omit the superscript index (σ) in the following where it is not explicitly needed, and restrict ourselves to a single component of the vector-valued FE space.
The matrix representation of this construction is with Kronecker products. Albeit triangles do not utilize the full potential of tensor product representations, this product form enables the BRIDGE code to make extensive use of sum factorization techniques, which are widely used in hp-FEM codes . For the application of derivative matrices we have, for example,
Ivrt⊗Dhrz,1coIvrt⊗Dhrz,2coDvrtco⊗IhrzĨu.
Here, Ihrz and Ivrt denote the Nh×Nh and Nv×Nv identity matrix, respectively. Ĩ denotes the interpolation from a finite element basis with nodes in the qhrz×qvrt quadrature points to an arbitrary (nodal) basis. The matrices Dhrz,1/2co∈Rqhrz×qhrz, Dvrtco∈Rqvrt×qvrt are defined as the gradient of Lagrange polynomials with nodes in the quadrature points. In practice, the multiplication with the interpolation matrix Ĩ does not count (it is shared with the calculation of the volume source term). Furthermore, the above equation describes derivative operations in a general finite element setup. Interpolation is trivial in the case of collocation, where the solution variables are stored at the nodes of the quadrature rule (often called Discontinuous Galerkin Spectral Element Method, DGSEM).
Similar formulations exist for other matrix-vector-multiply operations, e.g. integration. In terms of computational complexity, assuming n:=Nh=Nv=qhrz=qvrt for the number of degrees of freedom and the number of quadrature points, the number of multiplications and additions is reduced to 3n3+3n2 in contrast to n4 for a naive implementation.
Block-tridiagonal solve
The LSE has a block-tridiagonal structure and is directly solved with a generalized Thomas algorithm. At first, each such block is a (NσNvNh)×(NσNvNh) matrix. As can be seen both from the LSE coefficients (Sect. 3.1 in B21) and from the boundary conditions (Sect. 4.3 in B21), all these block matrices decouple in the horizontal directions if one uses collocation. We use Gauss-Legendre quadrature in the vertical and the Gauss-Legendre-like quadrature rules of on the triangle. Therefore, we consider Nh blocks consisting only of (NσNv)×(NσNv) matrices instead. For a 4th order scheme with Nh=10 base functions on a triangle this reduces the effort for an LU decomposition by a factor of Nh3=1000 and for the matrix-vector multiplication to calculate the new state in every RK stage by a factor of Nh2=100. Since the number of equations increases by a factor of Nh=10, the overall efficiency gain by using collocation is about a factor of 10.
“Sparse” Kronecker products
As an extension to the above Kronecker product implementation, both, the horizontal (explicit) and the HEVI-DG scheme in the BRIDGE code can exploit the fact that basis functions vanish on a cell face if the corresponding interpolation node does not reside on this face. Internal interpolation nodes in this case provide zero columns in the corresponding matrices.
In the horizontal discretisation not all polynomials have support on each face when the nodes are located on the boundary, for example, in a method with Gauß-Lobatto quadrature. This can be exploited to reduce the cost of face integration, which is otherwise the dominant cost of an explicit solver. There is, however, the disadvantage that in contrast to the Gauß-Legendre method numerical integration is only exact for polynomials of order up to 2N-1 when using a Gauß-Lobatto quadrature with N+1 nodes. To align both approaches, mixed quadratures have been proposed in the literature .
In the HEVI-DG scheme, there is another option to exploit the sparsity of the matrices in the Kronecker product: The HEVI formulation makes no assumptions on the vertical FE approximation, therefore we can use non-collocated finite elements in the vertical dimension. Choosing interpolation points suited for the evaluation of the surface integrals, the vertical FE basis is split into exterior functions {1,Nv} and interior functions {2,…,Nv-1}. We have Pn(±1)≠0 for exterior functions only in the coefficient matrices and may avoid calculation of zero entries during matrix assembly. Algebraically, the numerical solution of the block tridiagonal system
{Ãσσ′,B̃σσ′,C̃σσ′}
with Nh blocks of size NσNv can be solved in a computationally efficient manner as follows, see, e.g. for a similar application:
Reordering rows and columns into exterior and interior indices yields a block matrix
A=BCDEx1x2=f1f2,
where E (interior indices) is an easily invertible block diagonal matrix and B (exterior indices) is a block tridiagonal matrix with blocks of size n1:=2Nσ. We then solve the system with the Schur complement B-CE-1D:
B-CE-1D0DEx1x2=f1-CE-1f2f2.
This approach (static condensation) gets more efficient for higher order, i.e. where n1 is small.
The drawback of such a partly collocated FE discretisation is that interpolation operations and products with the mass matrix become slightly more expensive. We have
where Shrz and Svrt denote the values of all horizontal and vertical basis functions evaluated in the quadrature points. The (inexact) mass matrix, which is diagonal for discretely orthogonal basis functions, now reads in the case of horizontal-only collocation, when the quadrature points coincide with the node locations (“partly trivial interpolation matrices”, we have Shrz≡Ihrz):
M=(SvrtT⊗Ihrz)D(Svrt⊗Ihrz).
Remarks on <inline-formula><mml:math id="M289" display="inline"><mml:mi>p</mml:mi></mml:math></inline-formula>-adaptivity
One of the notable advantages of the DG approach is its ability to undergo p-refinement or p-coarsening. In other words, it can increase the polynomial degree p on elements to improve accuracy without altering the mesh. In this paragraph, we will briefly discuss the potential and limitations of the HEVI scheme in relation to p-adaptivity. Here, the focus is less on varying the polynomial order between different columns of prism elements. Instead, we consider discretizations in which different variables, or different layers within the same column, may be approximated by finite element components of different polynomial order.
Summarizing the above remarks, the evaluation of surface integrals is cheaper for DG with boundary-located nodes. However, apart from the issue of under-integration, this can be primarily exploited in the vertical direction: In the horizontal directions, the HEVI scheme requires collocated FE for computational efficiency. Besides, mass-matrix operations and interpolation are particularly cheap for collocated FE.
Decoupling of HEVI linear equation systems in the horizontal directions, i.e. collocation, implies an identical nodal finite element basis for the whole vertical column. This severe limitation for p-adaptivity is even further extended by the fact that individual low-order equation components (e.g. prognostic variables, auxiliary variables, tracer fields, etc.) would require additional interpolation. On the other hand, using the highest necessary quadrature order for different groups of equation components for all terms prohibits computational savings by low-order elements. Finally, heterogeneity between columns brings time-step restrictions for the explicit solver.
Consequently, a p-adaptive scheme is only computationally feasible in the vertical direction, where the HEVI-DG approach does not make assumptions about the finite element space. From a meteorological perspective, using a varying polynomial order in the vertical direction could be sensible, since the upper atmosphere contains much less water vapor than the troposphere below. Investigating this approach is left to future studies.
Boundary conditions in HEVI schemes using IMEX-RK methods
At least for budget equations of the type Eq. () (scalar case) or Eq. () (vector case) the following general statement can be made: “Boundary conditions determine the fluxes at the boundary”. However, in a HEVI discretisation or more general in an IMEX time integration method every flux is split into two partial fluxes
∂q∂t+divfex+divfim=Sex+Sim.
Thus, boundary conditions (BCs) only hold for the sum of the two fluxes (fex+fim)⋅n.
In particular for free-slip conditions, where several boundary fluxes (for mass, heat, tangential momentum, …) should vanish, B21 proposed to apply this “flux=0” condition for both flux parts (ex and im) independently and derived separate BCs for them. The motivation behind this was to have correct free slip conditions for every stage of an IMEX-RK scheme. This independent application seemed necessary, since the RK coefficients for a certain RK stage are different, in general, for the explicit and implicit part. This especially works fine for the (hyperbolic) Euler equations (i.e. without diffusion) since, by this procedure, the BCs for the explicit and implicit parts are identical. In contrast, the diffusive fluxes would in fact require different BCs for the explicit and implicit fluxes by this approach. Although it could be demonstrated in B21 that this leads to the desired vanishing of boundary fluxes, there are examples where the fields near the boundary show a non-physical spatially oscillating behaviour and the derivative variables do not approach the correct boundary values.
In fact it seems impossible to derive such BCs that produce the correct boundary condition for every RK stage in an IMEX-RK scheme. The only answer to this problem seems to abandon the requirement of getting the correct BCs for every stage: what is only required at the end of the day is the correct boundary behaviour for the final RK stage. This is much easier to fulfil: if the RK coefficients for the final IMEX-RK stage (the “corrector” step) are the same for the explicit and implicit part, bex=bim (we call this here the IMEX-RK “final stage condition”), then one can set the physically correct BCs separately both for the explicit and implicit fluxes, because in this case they will hold for the sum fex+fim, too.
There exist several IMEX-RK schemes in the literature that fulfil this “final stage condition”: e.g. SSP3(3,3,2), SSP3(4,3,3), ARK2(2,3,2), ARS2(2,3,2) and ARS3(2,3,3) (here we mention only those schemes inspected in ). However this property does not hold for the schemes Trap2(2,3,2), strong carryover UJ3(1,3,2), or ARS3(4,4,3). We note here, that this “final stage condition” is also mentioned in , where it is used to preserve linear invariants in IMEX-RK schemes.
The deeper reason why this works is the fundamental linear behaviour of the DG discretisation: the explicit and implicit flux parts (although they can be arbitrarily nonlinear) are included in the budget equation in a linear manner through the divergence term (Eq. ), the weak form essentially consists of linear scalar products, and all quadratures (together with the Jacobian) are linear operations of the function values and are done similarly for the explicit and implicit parts. Finally, in particular the Lax-Friedrichs flux combines the right hand and left hand physical (ex and im) fluxes in a linear manner.
Perhaps one additional remark: mostly BCs are set in a weak manner in a DG-scheme (as it is also done in B21), which treats the BC in the same approximation order as the solution of the PDE. Therefore, the variables do not immediately jump to their expected boundary values but relax towards them. Nevertheless, the BC itself (e.g. vanishing of certain numerical fluxes in a free-slip condition) is fulfilled in every time step with any of the above proposed IMEX-RK schemes that fulfil the “final stage condition”.
“flux<inline-formula><mml:math id="M301" display="inline"><mml:mo>=</mml:mo></mml:math></inline-formula>0” boundary conditions for the frictionless Euler equations (with energy)
The BCs for the free-slip (i.e. flux=0) conditions for equation set “E” in the inviscid case are determined as in B21 (Sect. 4.1.1 there): the LF flux for mass density vanishes when the reflection condition (in this section “L” means outside and “R” inside of a boundary)
ML3=-MR3
and the mirror condition
ρL=ρR
are fulfilled. Note that the latter also leads to vanishing numerical diffusion flux of the LF-numerical flux. Likewise the LF-flux momentum flux components T13 and T23 vanish with the additional mirror conditions
M1,L=M1,R,M2,L=M2,R.
These conditions also lead to mirror conditions for kinetic energy Ekin,L=Ekin,R and potential energy Epot,L=Epot,R. Therefore, once we have finally prescribed the mirror condition for total energy
EL=ER,
the condition for the remaining internal energy Eint,L=Eint,R follows directly. Since internal energy is only a function of pressure, the same applies to pressure, too, pL=pR, so that the LF-flux for total energy (with the physical energy flux ()) vanishes. To summarize: the only physical BC is M3=0 (discretised by the reflection condition ()) and there are no BCs for ρ, M1, M2, and E (i.e. they are discretised by the above mirror conditions). Of course, we have to express these BCs by our prognostic variables (contravariant unit components of momentum and deviations from the base state for ρ and E), which leads to a linear system of equations. This can easily be solved for the outside boundary values ..L as a function of the inside values ..R (similar to B21).
“flux<inline-formula><mml:math id="M323" display="inline"><mml:mo>=</mml:mo></mml:math></inline-formula>0” boundary conditions for the viscous Euler equations
Free-slip conditions for a viscous and heat conducting medium seem to be a contradiction, however, they are sometimes used in idealized test cases like the test; also see a few remarks in B21 about this point.
Now, we derive the BCs for the derivative variables, needed for the diffusive fluxes. From the BCs for the prognostic variables, given in the previous section, we can immediately derive kinematic conditions for their two horizontal covariant derivatives, namely
M;13=0,M;23=0,
and no conditions for the variables ρ;1,ρ;2,M1;1,M1;2,M2;1,M2;2, and furthermore no conditions for either E;1,E;2 or η;1,η;2 for equation set “E” or “ρΘ”, respectively.
In contrast to B21, we set no condition for ρ;3, because the continuity equation does not contain a diffusive mass flux contribution, and since under influence of gravity the hydrostatic equilibrium dominates, which clearly induces ρ;3≠0 at the ground. Nevertheless, we can conclude from Eqs. () and () that we also have
v;13=0,v;23=0.
From the diffusive momentum flux components, only the two horizontal components T13 and T23 must vanish. Since
Ti(D)k=-ρKav;ik+vi;k-ρKbδikv;jj,
the terms ∼Kb vanish for these two components, and with the above, we can directly conclude the two BCs
v1;3=0,v2;3=0,
or alternatively
M1;3=M1ρ;3ρ,M2;3=M2ρ;3ρ.
There is only one remaining momentum derivative variable M3;3 which obviously does not need to be set to fulfil any physical condition. In fact, a diffusion equation only allows setting one variable (here M3=0) but not additionally its derivative.
For the Euler equation set “ρΘ”, we can conclude from Eq. () that the vertical diffusive heat flux fϑ,D3 vanishes, if the BC
ϑ;3=ϑρ;3ρ
holds.
For equation set “E” we still have to define a BC for E;3 for a vanishing diffusive energy flux fE,Dk3+fE;Dh3 (Eqs. and ). First, fE,Dk3=vjTj3 already vanishes by the former BCs due to v3=0 and T13=0,T23=0. Second, from the requirement fE,Dh3=0, we can derive from its form Eq. () a BC for the vertical derivative of the inner energy
Eint;3=cpcvEintρ;3ρ.
The boundary behaviour of the other two energy contributions Ekin;3 and Epot;3 can be directly derived from the former BCs. We get from the fact that there are no BCs for ρ and ρ;3 that
Epot;3|L=Epot;3|R.
To derive the conditions for the kinetic energy derivative we have to express several co- and contravariant components and derivatives by the above BCs (e.g. M3=(M3-g31M1+g32M2)/g33) and finally get
Ekin;3=Ekinρ;3ρ.
In total we get the BC
E;3=Ekinρ;3ρ+Epot;3+cpcvEintρ;3ρ.
As in the previous section we formulate these BCs as reflection or mirror conditions, e.g. from Eq. () we get the reflection condition
EL;3=-ER;3+2Ekinρ;3ρ+Epot;3+cpcvEintρ;3ρR.
This results in an equation system between the inner (R) and outer (L) variables. In the free-slip case this equation system is linear and can be solved (at least numerically) easily for the outer variables.
A final remark: we have assumed in the previous derivation that the diffusion coefficients are set by a mirror condition KH,L=KH,R, KM,L=KM,R.
Linearisations of the BCs for the HEVI scheme
For the vertically implicit solver we still need a linearisation of these BCs. This is relatively easy for most of the variables, we just demonstrate it here for the most complicated variable, the vertical derivative of the energy. To this purpose we write its BC (Eq. ) in the BRIDGE variables leading to
E;3=g3k1-cpcv121ρgi′j′Mi′Mj′ρ;kρ+1-cpcvρ;kΦ(r)+ρΦ;k+cpcvEρ;kρ.
The following are all derivatives by the prognostic variables (ρ, Mk′, E) and the covariant derivative variables:
∂E;3∂ρ=g3kΦ;k-cpcvEg3kρ;kρ2,∂E;3∂ρ;3=1-cpcvg33Φ+cpcvEg331ρ,∂E;3∂Mk′=0,∂E;3∂M;3k=0,∂E;3∂E=cpcvg3kρ;kρ,∂E;3∂E;3=0.
In contrast to the flux and source term linearisations, for the BCs we found a linearisation only around the reference state to be sufficient.
Filtering
In strongly nonlinear problems some sort of regularization is needed for higher order methods. This holds in particular for DG methods, as emphasized by “... DG methods tend to demonstrate remarkable instabilities, ...” p. 407, and “Spectral element models typically use some kind of filtering because this method, like the spectral method, is not immune to aliasing errors that arise from the nonlinear terms” . One widespread and computationally cheap method is filtering Sect. 5.3, i.e. one tries to damp amplitudes for higher polynomial degrees (i.e. low pass filtering). This is done in a “postprocessing step” after each (IMEX-)RK step (as done e.g. in ). Therefore filtering itself has no stability issue. Filtering works best, i.e. the low pass properties are fulfilled correctly, when an orthogonal modal base ϕl(m)(r) is used. This means to transform the amplitudes from the original polynomial (nodal or modal) base into this orthogonal modal base Qk(m)=∑jTkjQj, to apply filter coefficients Ql(m)filt=∑kFlk(m)Qk(m), and to transform back, resulting in the final filter matrix F=T-1⋅F(m)⋅T (e.g. ).
Of course, such a postprocessing filter step must not destroy the local conservation properties of the DG scheme. This is obviously the case, if the lowest modal base function ϕ1(r)=const., and the related filter coefficient F11=1. The local conservation holds since due to the assumed orthogonality, all other base functions are “massless”. Orthogonality is always defined by a scalar product, and it is important to use just the scalar product that is induced by the weak form (Eq. ) of the DG method, i.e. that is induced by the metric density (or the Jacobian) J≡g, therefore
(f,g)=∫VdvJ(r)f(r)g(r)
or its discrete analogue using a quadrature rule. Since the metric density is spatially varying in general, during the initialization phase of the simulation such an orthogonal base is determined independently for each grid cell by a Gram-Schmidt orthogonalisation procedure. Then the filter matrix F is determined for each grid cell, and the simulation can be started.
For practical reasons, we employ an exponential filter of the form
Fjj(m)=e-αη2s,η=j-NcN,
to define the filter coefficients F(m) in the orthogonal modal space; see, e.g., the classical reference on filters for spectral approximations and for applications to DG methods.
N denotes the spatial order of the DG scheme (i.e. maximum polynomial degree plus one). The other constants α, Nc, s must be chosen depending on the problem. If one roughly relates a polynomial with a certain degree and a Fourier component with the same number of zeros, then in particular for Nc=1 one can derive a certain similarity between this exponential filter and a hyper-diffusion operator of the form ∼α∇2s.
It should be emphasized that Fjj(m) is not a spectral (idempotent) projector with sharp modal truncation. Its effect depends on the frequency with which the filter is applied, which makes a time-step-independent interpretation more difficult than for other stabilization approaches, such as hyper-diffusion. Nevertheless, this simple spectral filter preserves the relevant conservation property while selectively targeting the highest modes, and we consider it sufficient for the present purpose.
Another remark should be made about the order of the scheme. If a filter is used with F11(m)=F22(m)=…=Fnn(m)=1 and Fll(m)<1 for l>n then the whole DG scheme is limited to order n. Therefore, formally an exponential filter does not retain high order. However, if the above requirement is not fulfilled, one could let converge all filter coefficient towards 1 according to Δx in a suitable manner (analogous to the case of hyper-diffusion, where one can retain convergence by setting the numerical diffusion coefficient α proportional to a suitable power of Δx).
In the test cases of Sect. and (both using the inviscid Euler equations and mainly dealing with a baroclinic instability) it turns out that additionally to an exponential “base” filter, which is applied in every grid cell, one should use a stronger filter in selected grid cells that are “marked” by a so called oscillatory sensor, also called discontinuity sensor or smoothness indicator . This sensor is calculated as the ratio between the sum of the squares of those amplitudes related to the highest polynomial degree and the sum of the squares of all amplitudes (i.e. again one transforms to an orthogonal base, but for this sensor there is no need to consider the metrics). Therefore, unrealistic and suspicious oscillations deliver relatively high values of the sensor.
Finally we want to make a general remark about filtering in DG schemes for atmospheric flows. Since filtering is a post-processing step outside of the DG time step, it inevitably disturbs the highly important balancing between the buoyancy term and the pressure gradient. We have found that a too strong filtering is therefore not only detrimental for accuracy but also for numerical stability. To some extent this additional constraint limits the effectiveness of filtering in DG schemes for atmospheric flows compared to other fluid simulations where well-balancing is of less importance.
Numerical results
To demonstrate the applicability of the presented DG solver we will carry out a number of idealised test cases, which have been set as a certain standard in the validation of dynamical cores for atmospheric models. For some of these tests we additionally need upper and/or lateral damping layers to prevent non-physical wave reflection at artificial and therefore non-physical boundaries. Here we use the damping layer formulations given in B21. Some test cases use a vertical grid stretching. This is included in the terrain-following coordinate () and we use the quadratic stretching function f described in B21, too. Although the BRIDGE code also allows Gauß-Lobatto quadratures (see Sect. ), for the following tests we only use Gauß-Legendre quadratures.
Stationary 1D (vertical) flow
The first striking difference between the two variants of the Euler equations can be seen in one of the simplest test cases: the simulation of the Euler equations in an only 1D vertical column setup with constant grid spacing Δz and very simple initial conditions. In the setup the model top lies at Htop=10km, Δz=1km. The atmosphere is initialized isothermally (i.e. height independent temperature) with T=250K and at rest (one can choose a non-vanishing horizonal base velocity u just to check the correct behaviour of the free slip boundary condition, but this does not influence the otherwise purely vertical dynamics of this test). The reference state is also chosen isothermal with T0=200K (Eq. ) (it is only important that it is not identical with the initial state, of course, larger deviations between these two values will lead to larger signals). The matrices for the HEVI solver are updated every 50 time steps; related jumps in the solutions can be recognized in the time series in Fig. . Using smaller values (down to 1) only has a small influence on w and an almost negligible influence to T (not shown). All simulations have been performed with the SSP3(3,3,2) IMEX-RK scheme and a quite small time step Δt=0.2s.
Figure shows snapshots of w and T for the equation set “ρΘ” and for a 4th order DG scheme at several output times. One recognizes the increase in the amplitudes; the simulation crashed several hundred time steps later. To reduce this problem, and proposed the so-called source term filtering (STF) (see B21 for the implementation in the HEVI approach). Using STF avoids the linear instability: the vertical velocity w does not grow exponentially any more; in fact, it remains at least at a constant perturbation level. However, STF still leads to an almost linear temporal growth in temperature perturbations that can lead to a model break (at least for 3rd and 4th order) after several 100 000 time steps due to a nonlinear instability mechanism. This can be seen in Fig. , which shows the maximum absolute values of w (top) and temperature T (bottom) over time for both equation sets (“ρΘ” left, “E” right), different spatial approximation orders (3,4,5) and with or without STF.
1D vertical Euler equations for Euler equation set “ρΘ” with 3rd degree polynomials and without source term filtering (STF). Solutions after 1, 10, and 100 time steps (top, middle, bottom row). Left: vertical velocity w, Right: temperature T. The dots denote the solution on the cell quadrature points.
In contrast, using the Euler equations with E (again first without STF) leads to simulations where w perturbations in fact tend towards 0 after several thousands of time steps. Consequently this does not lead to an increase in temperature perturbations and the simulation remains stable, at least for all considered orders 3, 4, and 5.
Running the Euler equations using Ewith STF has no influence on stability, and leads to quite similar results after 100 000 time steps compared to the runs without STF. However, it can lead to slightly larger amplitudes in w and T during the first time steps. In contrast, later on (after several 100 time steps) STF can lead to a quicker convergence of w towards 0.
These simulation results are in good agreement with a normal mode stability analysis given in Appendix . The linear instability arising in the “ρΘ” system in fact can be resolved by STF over a large range of temperature stratifications. However, the equation set “E” behaves closer to the physical stability/instability behaviour, whereas the “ρΘ” set does not.
We have checked (not shown) that proper convergence is, of course, achieved in the case of stable simulations (i.e. in the case of the Euler equations using E), which means that the perturbations in w and in T are reduced by the appropriate order when reducing Δz and Δt, i.e. well-balancing problems do not violate convergence, as expected.
1D vertical Euler equations. Top: maximum absolute value of vertical velocity w over timestep, bottom: maximum value of temperature T over timestep. Left: Euler equation set “ρΘ”, Right: Euler equation set “E”. Shown are values for spatial DG order 3, 4, and 5 and with STF (lines denoted with “..._filt”) or without STF.
If not mentioned otherwise, the following simulations with equation set “ρΘ” always use STF whereas no STF is used for equation set “E”. Also we will use horizontally a 4th order (i.e. polynomial degree 3) DG spatial discretisation; this is the lowest order that reduces grid imprinting problems in shallow water equations to an acceptable level . Vertically we will use a 5th order scheme (i.e. polynomial degree 4). This is motivated by the results of Fig. 2, where the overall well-balancing effects are acceptably small.
Flow over mountains – linear case
To demonstrate the correctness of the terrain-following coordinate formulation, we consider a 2D flow over very low mountains on a flat plane, for which a linearised analytic solution exists (, here we use the version from ). The setup is analogous to B21 (see Sect. 5.2 for details): a couple of mountains with a maximum height of 10 m are set in a stably stratified atmosphere with constant Brunt–Väisälä frequency N=0.01s-1, with constant inflow velocity of u0=10ms-1, and without Coriolis force. We use (2⋅140)×16 grid cells (where one Δx-interval comprises two triangles, i.e. we use the quasi-2D methodology for our triangle cell stripe of Appendix ), horizontal grid spacing is Δx=1km, the vertical grid is stretched. Figure shows that the simulation of both equation sets agree quite well with the analytic solution near the ground. As it can be expected, the damping layer reduces the simulated vertical velocity near the model top. We used a time step of Δt=0.2s for equation set “ρΘ”, whereas equation set “E” tolerates a slightly larger time step of 0.25s (to be more precise: set “ρΘ” runs stable for Δt until 0.23s, whereas set “E” runs stable until 0.26s).
Linear 2D flow over mountain test. Vertical velocity w=v3′′ after t=24 h. Black lines: analytic solution, shaded and grey, dashed lines: simulation. Left: Euler equation set “ρΘ”, Right: Euler equation set “E”.
Flow over mountains – steep case
To demonstrate the quite stable behaviour of the DG method even over very steep terrain, we consider a similar test setup as in B21 (Sect. 5.5.1, see some further setup details there). A couple of mountains with a maximum height of 8km and a very steep maximum slope angle of about 72° is set into an atmosphere with inflow velocity u=10ms-1, a stable temperature stratification with a constant Brunt–Väisälä frequency of N=0.01s-1, and without Coriolis force (Ω=0). We again use the quasi-2D methodology for our triangle cell stripe of Appendix , with a horizontal grid spacing Δx=4 km and a vertical grid stretching. Such steep and high mountains generate a non-stationary solution due to gravity wave breaking, see Fig. after 23 and 24 h of simulation time. In reality this wave breaking generates turbulence that damps it. Therefore, a simple Prandtl mixing length turbulence model () is used with a height-dependent (more precisely, z is the vertical distance from the bottom orography) turbulent length scale
lt=κz1+κzlB,
where we have chosen a constant Blackadar length scale lB=40m and the von-Kármán-constant κ=0.4. No additional filtering was applied. We note that B21 used a Smagorinsky model, whereas here we decided for the Prandtl model since it seems to generate slightly more realistic ranges for the diffusion coefficient KM=KH, see Fig. , bottom row (negative values of KM are an inter-/extrapolation artifact of the plotting tool, in the BRIDGE code any possible negative value is clipped away).
2D flow over steep mountains. Top: horizontal velocity component u (shaded) and potential temperature Θ (isolines) after t=23 h, middle: the same after t=24 h, bottom: diffusion coefficient KM (shaded) and potential temperature Θ(isolines) for t=24 h. Left: Euler equation set “ρΘ”, Right: Euler equation set “E”.
Both equation sets “ρΘ” and “E” roughly show a similar qualitative behaviour (Fig. , left and right, respectively). The main difference is a more pronounced lee wave pattern in equation set “E”. Note that a closer similarity between both solutions cannot be expected in such a chaotic flow regime.
Density current
A strongly nonlinear 2D test case simulating a falling cold bubble was defined by . When the bubble hits the ground and evolves in both horizontal directions, Kelvin-Helmholtz instability generates several vortices. The simulation is diffusion-limited by a constant diffusion coefficient KM=KH=75m2s-1. The BRIDGE setup uses a grid spacing of Δx=Δz=400m (again we use the quasi-2D methodology of Appendix ), with a time step of 0.06 s using the SSP3(3,3,2) scheme for the Euler equations with diffusion terms. This results in horizontal and vertical Courant numbers of C=(c+|v|)ΔtΔxi≈0.05. Otherwise, remarks given in B21, Sect. 5.4, about boundary conditions and details about the diffusion term also apply here. The reference state is given by Eq. (). Figure shows the potential temperature after 900 s, which for both equation sets is in a good agreement with . Note, that our grid spacing is 16 times larger than the benchmark simulation and demonstrates the benefit of using higher order methods just for diffusion-limited flows. The total mass in the whole domain is neither increasing nor decreasing during the simulations, i.e. the jumps are below machine precision. Total energy (for the equation set “E”) in the domain is slightly increasing with a relative change of +7×10-14 during the simulation (i.e. over 15 000 time steps). The relative changes in the domain integral of ϑ (for equation set “ρΘ”) are also below machine precision.
Density current, cold bubble test. Potential temperature Θ after 15 min. Left: Euler equation set “ρΘ”, Right: Euler equation set “E”. The contours are the same as in Fig. 1 of .
Linear sound and gravity wave expansion on the sphere
To demonstrate the proper convergence properties of the BRIDGE code, an analytical solution is needed. present a linearised analytic solution for the compressible, non-hydrostatic Euler equations for the expansion of sound and gravity waves in a spherical shell of thickness H=10 km around a “small earth” with radius 6371.229/50km. These waves are excited by a weak warm bubble set at the north pole. The simulation is performed using various icosahedral grids with an average grid spacing of 5.681, 2.840, 1.420, 0.710°, and 3, 6, 12, 24 vertical grid cells, respectively (in the ICON nomenclature: an R2B2L3, R2B3L6, R2B4L12, R2B5L24 grid, the grid spacing is identified as the average square root of all triangle areas on the sphere). The further setup is described in for test scenario (A) with one exception: here we use a temperature perturbation ΔT=0.001K that is smaller by a factor of 10. The reason for this is the strong reduction of errors in a 4th order DG scheme, which otherwise would exhibit visible deviations from the proper convergence due to non-linear contributions in the Euler equations. To calculate the error measures, the BRIDGE output is compared with the analytic solution on every quadrature point in the 3D spherical shell. The analytic solution itself was precalculated on a structured grid on a latitude-z-plane and is then linearly interpolated to every quadrature point (this structured grid is so highly resolved that no detrimental effect on the convergence rate by this linear interpolation is visible). Figure shows error measures for vertical velocity w=v3′′ and temperature T after 75 min of simulation time. In general, we expect 4th order convergence since we use horizontally 3rd order polynomials. Although for the time integration the only 3rd order SSP3(4,3,3) scheme was used here, we think that the time step is small enough that the overall convergence rate is determined by the spatial discretisation instead of the temporal discretization. For equation set “E” the L1 error for w in fact shows exact 4th order convergence, whereas L2 and even more L∞ are not yet in their convergent regime (note that these fully 3D simulations quickly become very time consuming for higher resolutions). In contrast, equation set “ρΘ” shows much higher errors for w for coarse (vertical) resolutions than set “E”, as can be expected by the results of Sect. . This might be the reason why one seemingly sees a higher than 4th order convergence. The temperature even seems to converge in 5th order for both equation sets. We conclude that the vertical discretisation order dominates the convergence behaviour of the temperature field, in contrast to the vertical velocity field, which seems to be dominated by the horizontal convergence order.
Linear sound/gravity wave test on the sphere, L1, L2, L∞ of vertical velocity w (top) and of temperature (bottom). Left: Euler equation set “ρΘ”, Right: Euler equation set “E”. The grey, straight lines denote 4th order convergence behaviour; dashed lines denote 5th order.
Figure shows a work precision diagram, i.e. the L2 error of the vertical velocity against wall clock time, both for BRIDGE and for the ICON model . All these runs have been performed on AMD EPYC 7502 32-Core Processors with 2.5 GHz (which are the current so-called login nodes at the DWD), BRIDGE mostly runs on 192 processors (384 processors for the highest resolution) and ICON on three different processor configurations for the grids R2B5L12, R2B6L24, R2B7L48. All wall clock times are converted as if running on 192 processors. In this test BRIDGE obviously outperforms ICON by far. However, this is of course due to the very smooth fields that only occur in this test case. In other words, higher order schemes necessarily must outperform the standard second order schemes in this test case.
Work precision diagram for the linear sound and gravity wave test on the sphere: L2 error of vertical velocity against the analytic solution vs. wall clock time. Blue line: equation set “E”, red: equation set “ρΘ”, black: ICON.
Baroclinic instability testThe Jablonowski, Williamson (2006) test setup
The baroclinic instability is one of the most important large-scale dynamical mechanisms in the atmosphere and to a large extent determines the weather in the mid-latitudes. Several idealized test setups have been proposed to test this phenomenon on the sphere; here we use the setup of : two purely zonal jet streams are analytically prescribed together with the temperature stratification in pressure coordinates (i.e. for a hydrostatic approximation of the Euler equations). To apply it for the compressible Euler equations we use the iteration procedure described in the appendix of . The baroclinic instability is triggered by a small perturbation in the zonal velocity component u.
One important goal in new dynamical core developments is the increase of both accuracy and efficiency (other goals are good numerical stability, and so-called mimetic properties like conservation). Therefore, as in the density current test before, one may ask, what is the coarsest possible grid spacing that is still in good agreement with the benchmark result. For a (horizontally) 4th order scheme one may expect that an about 4 times larger grid spacing can be used compared to standard second order solvers. Here, BRIDGE simulations are done for a horizontal grid with an average grid spacing of about 315 km (R2B3 grid in the ICON nomenclature). Note that this is an about 5 times larger grid spacing (i.e. a bit larger than estimated) than used by the benchmark models in Fig. 7. 10 equidistant vertical levels are used until a model top of 40 km. A time step of Δt=80s was used, which is mainly determined by the horizontal grid spacing, resulting in a Courant number of Cx=(c+|v|)ΔtΔx≈0.1. The linearisation coefficients of the implicit solver are updated every 10 time steps.
A first striking difference in the two Euler equation sets “ρΘ” and “E” can be seen if we apply the BRIDGE code without filtering. Figure shows the meridional (north-south) velocity component v after 30 h. The simulation using equation set “ρΘ” (with STF) generates large perturbations in v that grow exponentially until a model crash after about 2.7 d. These mostly vertically oscillating perturbations seem to be an artifact of the above described well-balancing problem. In contrast, equation set “E” does not show such problems and remains stable over about 7.5 d.
Baroclinic instability test, no filtering. Meridional velocity component v after t=30h. Left: Euler equation set “ρΘ”, right: equation set “E”.
However, these results completely change if we apply filtering. We first apply a base filter with α=1.0 (horizontal) or α=0.5 (vertical), Nc=1 and 2s=10; this damps the amplitudes for the highest polynomial degree by about 5 % per timestep. Now the simulation using equation set “ρΘ” remains stable over more than 32 d and the results are quite close to what is shown in Fig. , left, which is in good agreement with the benchmark results of Fig. 7. This demonstrates that this filter smoothing is acceptable and that the chosen grid is sufficiently fine. However, the runs with equation set “E” only gain one day more and then crash after about 8.5 d. This larger sensitivity against instabilities for set “E” is also mentioned in and is explained there with its fundamental non-dissipativeness. The instability occurs at the ground where the horizontal velocity fields tend to increase in a non-physical manner.
Now we additionally apply the oscillatory sensor (based on density ρ̃) with a relatively low threshold of 0.0005. If this threshold is exceeded in a grid cell, then the stronger filter with α=0.5 (horizontal) or α=0.4 (vertical), Nc=1 and 2s=4 is applied there; this damps the amplitudes for the highest polynomial degree by about 15 % per timestep. With these filter values the simulations for both equation sets, Fig. , agree quite well with the benchmark results of Fig. 7 for both temperature in 850 hPa and surface pressure. However, whereas the run for equation set “ρΘ” remains stable over at least 40 d, the simulation for equation set “E” breaks after about 18 d. Better stability properties of this run would require stronger filtering and therefore would reduce the accuracy. This seems to be an indication that equation set “E” suffers more from nonlinear instabilities than set “ρΘ”.
Baroclinic instability test, global conservation properties of mass (red) and either ρΘ or total energy (blue). Left: Euler equation set “ρΘ”, with filtering. middle: set “E”, with filtering. right: equation set “E”, without filtering.
Figure shows the relative changes of globally integrated mass and either ρΘ or total energy over time, i.e. the property (It-I0)/I0, where It is the global integral of one of these properties at time t. The relative total mass change in the whole domain is about -7×10-16 per simulation day. The relative changes of total energy or total ρΘ are roughly in the same order of magnitude. These are quite small changes; e.g. in a 1000 year climate simulation it would mean a relative change of global mass of the order -2×10-10, which is so small that it wouldn't be detectable by any observation in a real atmosphere. However, this result is in contradiction to the assumed local (and the resulting global) conservation property of a DG scheme mentioned in Sect. . In the right panel in Fig. the filter is disabled, and in fact now there is no linear trend anymore neither in total mass nor in total energy, but only statistical noise at the roundoff limit. Therefore, we conclude that these losses do not occur due to the DG solver itself but due to tiny conservation violations of the filtering. One can speculate that these may come from roundoff errors in the Gram-Schmidt orthogonalization procedure described in Sect. .
The simulation took about 623 s per simulated day for both equation sets on 32 processors (2 nodes with 16 processors each) on the computer mentioned in Sect. . The simulations used 5120×10=51200 grid cells with 10⋅5=50 base functions for each grid cell. This means 50 quadrature points in the cell and 3⋅4⋅5+2⋅10=80 quadrature points at all prism faces. For the 5 Euler variables this means in total 51200⋅50⋅5=12.8×106 degrees of freedom, calculated on 51200⋅(50+80)=6.656×106 quadrature points. This wall clock time is comparable to those given for the “Finite-volume” model with comparable resolution given in Table 4 in on 32 processors on an IBM power 4 architecture. However, ICON runs on the same architecture as we used a factor of about 6 faster for a similar effective resolution (a R2B5L40 grid).
Baroclinic instability test, with filtering. Top: temperature T in 850 hPa at t=9d, bottom: surface pressure ps. Left: Euler equation set “ρΘ”, right: equation set “E”. The colors are similar to those used in Fig. 7.
The Ullrich et al. (2014) test setup
have proposed a slightly different setup for the baroclinic instability that is based on steady analytic solutions by for setting the initial state. Therefore, this setup is much easier to implement in non-hydrostatic, height based dynamical cores than the hydrostatic setup of . In particular, there is no need to set an orography. Therefore, we briefly show results for this test, too. The grid spacing, time step and numerical settings are the same as in Sect. . In particular with these filter settings the run with equation set “ρΘ” remains again stable over at least 40 d, whereas the run with equation set “E” crashes after slightly more than 11 d. Nevertheless the comparison of temperature in 850 hPa and surface pressure in Fig. agree well for both equations sets with the benchmark results shown in .
Baroclinic instability test by Ullrich et al. (2014), with filtering. Top: temperature T in 850 hPa at t=8d, bottom: surface pressure ps. Left: Euler equation set “ρΘ”, right: equation set “E”. The isolines and colors are similar to those used in Figs. 4, 5.
Simplified climate run
The simplified climate run only uses the inviscid Euler equations (without moisture and without any other parameterization like turbulence or radiation) and just adds simple relaxation terms on the rhs of the equations. These terms relax towards a prescribed temperature field and the velocity back to steady state. Consequently, internal energy is supplied from outside and kinetic energy is constantly taken out in a way that resembles an averaged climate state of the earth (therefore no global energy conservation can be expected here).
Here an R2B2L8 grid, i.e. a triangle grid with about 630 km grid spacing and vertically 8 grid layers has been used. Now a vertically quadratic grid stretching was applied so that the first quadrature point is in a height of only about 32.8 m above ground (the thickness of the lowest grid cell is Δz=1060m) and the highest quadrature point is 319.0 m below the model top of 30 km. The time step is Δt=30s, leading to Courant numbers horizontally of Cx=(c+|v|)ΔtΔx≈0.02 and vertically of Cz=|w|ΔtΔz≈3301060≈0.1 (for the explicit part of the HEVI solver). We expect this value for Cz as stable with an SSP3(3,3,2) IMEX-RK scheme.
The filtering for equation set “ρΘ” is identically to what was used in Sect. . However, to run a stable simulation over 1200 d for equation set “E” a stronger base filter with values α=1.9 (horizontal) or α=1.0 (vertical), Nc=1 and 2s=10 had to be used. Additionally the update frequency of the linearisation coefficients had to be increased to every 10th time step (every 50th for equation set “ρΘ”).
Figure shows the zonally and temporarily averaged zonal velocity, which can directly be compared with Fig. 2 of . The main structures, in particular the strengths and lateral and height positions of the two jets, are in a good agreement with this reference. However, the result for equation set “E” (right) even seems a bit closer to Fig. 5 of , who also used the Euler equations with total energy. Slight deviations from the north-south-symmetry possibly can be traced back to the sampling strategy (note that the initial state and the relaxation fields have the full north-south-symmetry): the simulation produced output every 6 h, from which the temporary average was constructed. The relative total mass change is about -2.1×10-13 yr−1.
Simplified climate run. Zonal mean and temporal mean (over days 200 to 1200) of zonal velocity (in ms-1). Left: Euler equation set “ρΘ”, Right: Euler equation set “E”. The contours are chosen as in Fig. 2 of .
Conclusions and outlook
The main goal of this article is the comparison of two Euler equation sets using density and momentum, and either density weighted potential temperature ϑ=ρΘ or total energy E as prognostic variables, both in a numerical framework of a classical DG scheme with the HEVI approach using IMEX-RK time integrators. Whereas the formulation with ϑ is mainly described in B21, here the details of a formulation with E are explained, in particular for the use in an IMEX-RK scheme. Furthermore, the combination of the HEVI approach with the local and terrain-following coordinate to treat atmospheric flows on the whole sphere are described.
One important goal of the resulting BRIDGE code is to achieve local conservation of the prognostic variables. Consequently, the use of E seems to be favourable because it especially means to fulfil the first law of thermodynamics. In contrast, although ϑ is related to entropy just under the conditions of this article, it does not mean to fulfil the second law of thermodynamics under more general conditions e.g. using moisture. In fact, some of the test cases demonstrate superior well-balancing properties of the Euler equations using E: in particular in the simple 1D vertical test, Sect. , together with the related normal mode analysis given in Appendix and in the linear sound and gravity wave expansion test, Sect. , demonstrating lower errors in particular for coarser resolutions. In particular, no accuracy reducing STF is needed for the equation set “E”. Additionally, slight advantages could be seen in the flows over mountains – a slightly larger time step could be used in the linear flow over mountains, see Sect. , and perhaps a slightly more realistic flow pattern in the flow over steep mountains, Sect. , could be achieved (although the true solution is not known in this case).
On the other hand, the equation set “E” suffers more from non-linear instabilities than equation set “ρΘ”, seen in the baroclinic instability test in Sect. and the simplified climate run in Sect. . This is probably a result of the non-dissipative property of an energy-conserving scheme . In both cases it was harder to find a setup that runs stable for set “E”.
We want to emphasize that we only used filtering as a stabilization mechanism, mainly because this is a computationally quite cheap method. As one of its main drawbacks is the violation of well-balancing properties if applied to the atmosphere. For DG schemes there exist several other stabilizing or dealiasing techniques (for a survey see e.g. the book of ). For example, the artificial viscosity method is widespread. One rather technical reason for not applying it is that we did not want to use the more expensive BR1 scheme in these (essentially frictionless) test cases. Recently, hyper-diffusion has been proposed for element-based Galerkin schemes by , a method that is used in many atmospheric models (like ICON). In a DG scheme the hyper-diffusion term can be incorporated in two ways. One option is to extend a scheme such as BR1 by introducing auxiliary variables for the higher-order derivatives; this approach is accurate but computationally expensive. Alternatively, the term can be added solely as a source term, which is computationally cheap but sacrifices strict conservation because the inter-cell fluxes are omitted.
Despite the somewhat mixed results for the inspected two equation systems, we will in particular continue the work on equation set “E”, due to their other positive properties described above and in the introduction. One reason for our optimism concerning numerical stability is the fact that every realistic application of an atmospheric model has to use turbulence parameterizations, which in many cases (though not generally) just perform a stabilization of the above mentioned model breakdowns. The other, more important, reason are recent developments for DG schemes towards entropy stable or even entropy conserving schemes that just build on set “E” and which have achieved a certain maturity . So it could be demonstrated that the baroclinic instability test of Sect. can be run without any model breakdowns occurring and without any other stabilization measures . Steps to extend these methods to triangles have been done by . However, at the moment it is less clear how to apply these new techniques to the combination of triangle grids, the HEVI approach, and the covariant formulation used in this article. In particular to the last point, recently developed an entropy stable DG scheme for the covariant formulation of the shallow water equations on the sphere. We plan to follow these new developments in the near future. Such provably stable approaches even could lead to a reduction of too excessive vertical velocities, which would allow to increase the time steps (see in particular the limitation given in Sect. ). We further will inspect alternative IMEX-RK schemes (e.g. those given in ) that might allow larger Courant numbers and therefore would lead to an improved efficiency of our BRIDGE code.
Quadrature rules for x-z-slice models on unit triangles
A few test cases in this article use a 2-dimensional “x-z-slice” model, i.e. we have only one horizontal direction. Since the BRIDGE code only can use triangle cells in the horizontal direction, practically, we use a stripe of squares, each square split into two triangles. Or, alternatively described, a stripe of two saw tooth rows (one upward and one downward oriented). Consequently, it is reasonable to use only 1D, x-dependent base functions in the horizontal direction for each triangle, and likewise only 1D quadrature rules. However, this requires definition of quadrature rules for only x-dependent polynomials not on a 1D interval, but on the 2D unit triangle D (-1≤x≤1, -1≤y≤-x). This results in a Gaussian quadrature rule of the form
∫Dpn(x)dxdy=∑i=1Nwipn(xi).
To this purpose, we generate 2N equations for the first 2N monomials pn(x)=xn, n=0,…,2N-1 by using
∫Dxndxdy=∫-11dx∫-1-xdyxn=2n+1n even,-2n+2n odd.
This highly nonlinear equation system for the quadrature nodes xi and weights wi is solved by a Newton iteration procedure. The resulting quadrature rules for different N are given in Table .
Gaussian quadrature rules for only x-dependent base functions on the unit triangle.
By an analogous procedure one can get Gauß-Lobatto rules for
∫Dpn(x)dxdy=∑i=1Nwipn(xi),x1=-1,xN=+1.
Although we have not used these in this article, for completeness the first quadrature rules for different N are given in Table .
Gauß-Lobatto quadrature rules for only x-dependent base functions on the unit triangle.
Nxiwi2-14/312/33-19/18-1/525/1812/184-18/30-0.54691816067802711.08570226039551580.26120387496374130.614297739604484211/305-125/150-0.70882014211432480.7943389359572047-0.13230082077732190.73600436948163330.50778762955831590.289656694561162012/150Normal mode analysis for the 1D (vertical) Euler equations
In this section the stability properties of the well-balancing issue are inspected. To this purpose we only consider one grid cell and restrict ourselves to the 1D vertical linearised Euler equations (i.e. using the linearisations in Sect. and ), without diffusion terms. So, for the equation set “ρΘ” we use
∂∂tρ̃Mzη̃+∂∂z01000∂p0∂η00η0ρ00ρ̃Mzη̃=000-g00000ρ̃Mzη̃,
and for the equation set “E”
∂∂tρ̃MzẼ+∂∂z010-Rcvgz0Rcv0E0+p0ρ00ρ̃MzẼ=000-g00000ρ̃MzẼ,
both with boundary conditions Mz(z=0)=0 and Mz(z=L)=0.
Maximum amplification factor for the 1D vertical stability problem. Left: Euler equation set “ρΘ”, Right: Euler equation set “E”. Top: without STF, Bottom: with STF.
In contrast to a von-Neumann-stability analysis, a normal mode analysis can treat linear equations with non-constant coefficients, too, e.g. vertically varying density, pressure or temperature. Therefore we can also inspect different temperature stratifications that we prescribe by a polytropic atmosphere of the form T(z)=T0+dTdzz. In Fig. the maximum amplification factors of both equation sets with or without source term filtering (STF) is shown over the stratification dTdz and over the time step. The maximum amplification factor is calculated as the maximum eigenvalue of the 12×12 amplification matrix for a 4th order DG scheme together with an SSP3(3,3,2) IMEX scheme (all above terms are treated with the implicit part) and a grid spacing of Δz=1000m. Note that the atmosphere becomes physically unstable for dTdz<-g/cp≈-0.01Km-1. We see that the equation set “E” behaves in a reasonable way: it is stable (the maximum amplification factor is equal to one) for the physically stable range and unstable for the physically unstable range. This is quite independent on the use of STF, so we don't need STF for equation set “E”. In contrast, the equation set “ρΘ” without STF is only stable in the physically unstable range and unstable in the physically stable range. Although the numerical instability is quite slow for moderate time steps (amplification factors are only slightly larger than one), it can nevertheless produce a problematic behaviour in longer simulation runs. STF resolves the numerical instability, with the exception of a very weak instability for larger time steps and dTdz<-0.005Km-1.
Code and data availability
The BRIDGE code together with all the scripts for running the test cases of this article is available under 10.5281/zenodo.17977588.
Author contributions
MB: implementation of equations, metrics, time integration; implementation, conducting, evaluation, and visualisation of test cases; numerical analysis; algorithmic concepts; preparing the manuscript. FP: software design; implementation; algorithmic concepts; preparing the manuscript.
Competing interests
The contact author has declared that neither of the authors has any competing interests.
Disclaimer
Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this paper. The authors bear the ultimate responsibility for providing appropriate place names. Views expressed in the text are those of the authors and do not necessarily reflect the views of the publisher.
Acknowledgements
We are grateful for several discussions about the topic with Gregor J. Gassner (Division of Mathematics, University of Cologne), Andrés M. Rueda-Ramírez (Universidad Politécnica de Madrid), Tristan Montoya (Institute for Aerospace Studies, University of Toronto), and Oswald Knoth (formerly at Leibniz Institute for tropospheric research, Leipzig). We would like to thank Francis X. Giraldo (Naval Postgraduate School) and another anonymous reviewer for their very thorough reviews and helpful hints and comments that definitely improved the article. We furthermore would like to thank Daniel Reinert (Deutscher Wetterdienst) for carrying out the ICON simulations for the linear wave expansion test case.
Financial support
This work was partly supported by the German Federal Ministry for Research, Technology and Space (BMFTR) project “ICON-DG” (grant 01LK2315C) of the “WarmWorld Smarter” program.
Review statement
This paper was edited by James Kelly and reviewed by Francis Giraldo and one anonymous referee.
ReferencesBaldauf, M.: A linear solution for flow over mountains and its comparison with the COSMO model, COSMO-Newsletter, 9, 19–24, https://www.cosmo-model.org/content/model/documentation/newsLetters/default.htm (last access: 20 July 2026), 2008.Baldauf, M.: Discontinuous Galerkin solver for the shallow-water equations in covariant form on the sphere and the ellipsoid, J. Comput. Phys., 410, 109384, 1–26, 10.1016/j.jcp.2020.109384, 2020.Baldauf, M.: A horizontally explicit, vertically implicit (HEVI) discontinuous Galerkin scheme for the 2-dimensional Euler and Navier-Stokes equations using terrain-following coordinates, J. Comput. Phys., 446, 110635, 10.1016/j.jcp.2021.110635, 2021.Baldauf, M. and Prill, F.: BRIDGE (Basic Research for ICON with Discontinuous Galerkin Extension), BRIDGE v0.9, source code, Zenodo [code], 10.5281/zenodo.17977588, 2025.Baldauf, M., Reinert, D., and Zängl, G.: An analytical solution for linear gravity and sound waves on the sphere as a test for compressible, non-hydrostatic numerical models, Q. J. Roy. Meteor. Soc., 140, 1974–1985, 10.1002/qj.2277, 2014. Bassi, F. and Rebay, S.: A High-Order Accurate Discontinuous Finite Element Method for the Numerical Solution of the Compressible Navier-Stokes Equations, J. Comput. Phys., 131, 267–279, 1997. Blaise, S., Lambrechts, J., and Deleersnijder, E.: A stabilization for three-dimensional discontinuous Galerkin discretizations applied to nonhydrostatic atmospheric simulations, Int. J. Numer. Meth. Fluids, 81, 558–585, 2016.Brdar, S., Baldauf, M., Dedner, A., and Klöfkorn, R.: Comparison of dynamical cores for NWP models: comparison of COSMO and Dune, Theor. Comput. Fluid Dyn., 27, 453–472, 10.1007/s00162-012-0264-z, 2012. Cernetic, M., Springel, V., Guillet, T., and Pakmor, R.: High-order discontinuous Galerkin hydrodynamics with sub-cell shock capturing on GPUs, Mon. Not. Roy. Astron. Soc., 522, 982–1008, 2023.Chaillat, S., Cottereau, R., and Sevilla, R.: A high-order discontinuous Galerkin Method using a mixture of Gauss-Legendre and Gauss-Lobatto quadratures for improved efficiency, hal-03695573, https://hal.science/hal-03695573 (last access: 20 July 2026), 2022. Chan: On discretely entropy conservative and entropy stable discontinuous Galerkin methods, J. Comput. Phys., 362, 346–374, 2018. Cockburn, B. and Shu, C.-W.: TVB Runge-Kutta Local Projection Discontinuous Galerkin Finite Element Method for Conservation Laws II: General Framework, Math. Comput., 52, 411–435, 1989. Cockburn, B., Lin, S.-Y., and Shu, C.-W.: TVB Runge-Kutta Local Projection Discontinuous Galerkin Finite Element Method for Conservation Laws III: One-Dimensional Systems, J. Comput. Phys., 84, 90–113, 1989. Cockburn, B., Hou, S., and Shu, C.-W.: The Runge-Kutta local projection Discontinuous Galerkin Finite Element Method for Conservation Laws IV: the multidimensional Case, Math. Comput., 54, 545–581, 1990. Gassner, G. J. and Winters, A. R.: A Novel Robust Strategy for Discontinuous Galerkin Methods in Computational Fluid Mechanics: Why? When? What? Where?, Front. Phys., 8, 1–24, 2021. Gassner, G. J., Winters, A. R., Hindenlang, F. J., and Kopriva, D. A.: The BR1 Scheme is Stable for the Compressible Navier-Stokes Equations, J. Sci. Comp., 77, 154–200, 2018. Giraldo, F. X.: A spectral element shallow water model on spherical geodesic grids, Int. J. Num. Meth. Fluids, 35, 869–901, 2001.Giraldo, F. X.: An Introduction to Element-Based Galerkin Methods on Tensor-Product Bases, Springer, ISBN 978-3-030-55068-4, 10.1007/978-3-030-55069-1, 2020. Giraldo, F. X., Restelli, M., and Läuter, M.: Semi-implicit formulations of the Navier-Stokes equations: aplications to nonhydrostatic atmospheric modeling, SIAM J. Sci. Comput., 32, 3394–3425, 2010. Giraldo, F. X., Kelly, J., and Constantinescu, E. M.: Implicit-Explicit Formulations of a Three-Dimensional Nonhydrostatic Unified Model of the Atmosphere (NUMA), SIAM J. Sci. Comput., 35, B1162–B1194, 2013.Giraldo, F. X., de Bragança Alves, F. A., Kelly, J. F., Kang, S., and Reinecke, P. A.: A performance study of horizontally explicit vertically implicit (HEVI) time-integrators for non-hydrostatic atmospheric models, J. Comput. Phys., 515, 113275, 10.1016/j.jcp.2024.113275, 2024. Held, I. M. and Suarez, M. J.: A Proposal for the Intercomparison of the Dynamical Cores of Atmospheric General Circulation Models, B. Am. Meteorol. Soc., 75, 1825–1830, 1994.Hesthaven, J. S. and Warburton, T.: Nodal Discontinuous Galerkin Methods, Springer, ISBN 978-0-387-72065-4, 10.1007/978-0-387-72067-8, 2008. Jablonowski, C. and Williamson, D. L.: A baroclinic instability test case for atmospheric model dynamical cores, Q. J. Roy. Meteor. Soc., 132, 2943–2975, 2006. Karniadakis, G. E. and Sherwin, S. J.: Spectral/hp Element Methods for Computational Fluid Dynamics, Oxford University Press, Oxford, ISBN 0198528698, 2005.Krais, N., Beck, A., Bolemann, T., Frank, H., Flad, D., Gassner, G., Hindenlang, F., Hoffmann, M., Kuhn, T., Sonntag, M., and Munz, C.-D.: FLEXI: A high order discontinuous Galerkin framework for hyperbolic–parabolic conservation laws, Comput. Math. Appl., 81, 186–219, 10.1016/j.camwa.2020.05.004, 2021.Kronbichler, M. and Kormann, K.: Fast Matrix-Free Evaluation of Discontinuous Galerkin Finite Element Operators, ACM Trans. Math. Softw. 45, 40 pp., 10.1145/3325864, 2017.Kronbichler, M., Kormann, K., Fehn, N., Munch, P., and Witte, J.: A Hermite-like basis for faster matrix-free evaluation of interior penalty discontinuous Galerkin operators, arXiv [preprint], 10.48550/arXiv.1907.08492, 2019. Landau, L. D. and Lifshitz, E. M.: Fluid Mechanics, Volume 6 of Course of Theoretical Physics, Pergamon Press, 2nd Edn., ISBN 978-0-08-033933-7, 1987.Landmann, B.: A parallel discontinuous Galerkin code for the Navier-Stokes and Reynolds-averaged Navier-Stokes equations, Ph.D. thesis, Universität Stuttgart, 10.18419/opus-3771, 2008. Läuter, M., Giraldo, F. X., Handorf, D., and Dethloff, K.: A discontinuous Galerkin method for the shallow water equations in spherical triangular coordinates, J. Comput. Phys., 227, 10226–10242, 2008. Lock, S.-J., Wood, N., and Weller, H.: Numerical analyses of Runge-Kutta implicit-explicit schemes for horizontally explicit, vertically implicit solutions of atmospheric models, Q. J. Roy. Meteor. Soc., 140, 1654–1669, 2014.Meister, A., Ortleb, S., and Sonar, T.: Application of Spectral Filtering to Discontinuous Galerkin Methods on Triangulations, Num. Meth. PDEs., 28, 1840–1868, 10.1002/num.20705, 2012. Michoski, C., Dawson, C., Kubatko, E. J., Wirasaet, D., Brus, S., and Westerink, J. J.: A Comparison of Artificial Viscosity, Limiters, and Filters, for High Order Discontinuous Galerkin Solutions in Nonlinear Settings, J. Sci. Comput., 66, 406–434, 2016.Montoya, T., Rueda-Ramírez, A. M., and Gassner, G. J.: Entropy-stable discontinuous spectral-element methods for the spherical shallow water equations in covariant form, J. Comput. Phys. 555, 114782, 10.1016/j.jcp.2026.114782, 2026.Orgis, T., Läuter, M., Handorf, D., and Dethloff, K.: Baroclinic waves on the β-plane using low-order Discontinuous Galerkin discretisation, J. Comput. Phys., 339, 461–481, 2017.Pareschi, L. and Russo, G.: Implicit-Explicit Runge–Kutta Schemes and Applications to Hyperbolic Systems with Relaxation, J. Sci. Comput., 25, 129–155, 10.1007/s10915-004-4636-4, 2005.Persson, P.-O. and Peraire, J.: Sub-cell shock capturing for discontinuous Galerkin methods, in: Collection of Technical Papers – 44th AIAA Aerospace Sciences MeetingConference Paper, 10.2514/6.2006-112, 2006.Reddy, S., Waruszewski, M., de Braganca Alves, F. A. V., and Giraldo, F. X.: Schur complement IMplicit-EXplicit formulations for discontinuous Galerkin non-hydrostatic atmospheric models, J. Comput. Phys., 491, 112361, 10.1016/j.jcp.2023.112361, 2023.Rueda-Ramírez, A. M., Ferrer, E., Kopriva, D. A., Rubio, G., and Valero, E.: A statically condensed discontinuous Galerkin spectral element method on Gauss-Lobatto nodes for the compressible Navier-Stokes equations, J. Comput. Phys., 426, 109953, 10.1016/j.jcp.2020.109953, 2021. Schuster, D., Brdar, S., Baldauf, M., Dedner, A., Klöfkorn, R., and Kröner, D.: On discontinuous Galerkin approach for atmospheric flow in the mesoscale with and without moisture, Meteorol. Z., 23, 449–464, 10.1127/0941-2948/2014/0565, 2014. Smith, R. B.: Linear theory of stratified hydrostatic flow past an isolated mountain, Tellus, 32, 348–364, 1980.Souza, A. N., He, J., Bischoff, T., Waruszewski, M., Novak, L., Barra, V., Gibson, T., Sridhar, A., Kandala, S., Byrne, S., Wilcox, L. C., Kozdon, J., Giraldo, F. X., Knoth, O., Marshall, J., Ferrari, R., and Schneider, T.: The Flux-Differencing Discontinuous Galerkin Method Applied to an Idealized Fully Compressible Nonhydrostatic Dry Atmosphere, J. Adv. Model. Earth Sy., 15, 10.1029/2022MS003527, 2023. Staniforth, A. and White, A. A.: Further non-separable exact solutions of the deep- and shallow-atmosphere equations, Atm. Sci. Lett., 12, 356–361, 2011. Straka, J. M., Wilhelmson, R. B., Wicker, L. J., Anderson, J. R., and Droegemeier, K. K.: Numerical solutions of a non-linear density current: a benchmark solution and comparisons, Int. J. Num. Meth. Fluids, 17, 1–22, 1993. Ullrich, P. A., Melvin, T., Jablonowski, C., and Staniforth, A.: A proposed baroclinic wave test for deep- and shallow-atmosphere dynamical cores, Q. J. Roy. Meteor. Soc., 140, 1590–1602, 2014.Vandeven, H.: Family of spectral filters for discontinuous problems, J. Sci. Comp., 6, 159–192, 10.1007/BF01062118, 1991.Waruszewski, M., Kozdon, J. E., Wilcox, L. C., Gibson, T. H., and Giraldo, F. X.: Entropy stable discontinuous Galerkin methods for balance laws in non-conservative form: Applications to the Euler equations with gravity, J. Comput. Phys., 468, 111507, 10.1016/j.jcp.2022.111507, 2022. Williams, D. M., Shunn, L., and Jameson, A.: Symmetric quadrature rules for simplexes based on sphere close packed lattice arrangements, J. Comput. Appl. Math., 266, 18–38, 2014. Zängl, G.: Extending the Numerical Stability Limit of Terrain-Following Coordinate Models over Steep Slopes, Mon. Weather Rev., 140, 3722–3733, 2012.Zängl, G., Reinert, D., Ripodas, P., and Baldauf, M.: The ICON (ICOsahedral Non-hydrostatic) modelling framework of DWD and MPI-M: Description of the non-hydrostatic dynamical core, Q. J. Roy. Meteor. Soc., 141, 563–579, 10.1002/qj.2378, 2015.