The efficient simulation of non-hydrostatic atmospheric dynamics requires time integration methods capable of overcoming the explicit stability constraints on time step size arising from acoustic waves. In this work, we investigate various implicit–explicit (IMEX) additive Runge–Kutta (ARK) methods for evolving acoustic waves implicitly to enable larger time step sizes in a global non-hydrostatic atmospheric model. The IMEX formulations considered include horizontally explicit – vertically implicit (HEVI) approaches as well as splittings that treat some horizontal dynamics implicitly. In each case, the impact of solving nonlinear systems in each implicit ARK stage in a linearly implicit fashion is also explored.
The accuracy and efficiency of the IMEX splittings, ARK methods, and solver options are evaluated on a gravity wave and baroclinic wave test case. HEVI splittings that treat some vertical dynamics explicitly do not show a benefit in solution quality or run time over the most implicit HEVI formulation. While splittings that implicitly evolve some horizontal dynamics increase the maximum stable step size of a method, the gains are insufficient to overcome the additional cost of solving a globally coupled system. Solving implicit stage systems in a linearly implicit manner limits the solver cost but this is offset by a reduction in step size to achieve the desired accuracy for some methods. Overall, the third-order ARS343 and ARK324 methods performed the best, followed by the second-order ARS232 and ARK232 methods.
Present-day global climate simulations typically use an atmospheric model configured with a horizontal resolution greater than 10 km. At these scales, the equations governing atmospheric motion can utilize the hydrostatic approximation, which assumes a balance between the gravitational and vertical pressure gradient forces and neglects terms related to vertical acceleration and transport of vertical momentum. As a consequence of this simplification, vertically propagating sound waves, which are of little significance in climate studies, are eliminated from the model. This practice is advantageous for computational efficiency with fully explicit time stepping methods, as vertical sound waves impose a stricter stability limit on step size than horizontal sound waves due to the high horizontal to vertical aspect ratio of the mesh. With the most significant constraint on step size removed, explicit approaches are an attractive option despite their step size limitations from horizontal sound waves. Explicit approaches are employed because of their ease of implementation, the locality of computations, and minimal parallel communication. However, in the near future, increased computational power will enable global climate simulations at scales beyond the hydrostatic limit where vertical acceleration cannot be ignored. At these high resolutions, new model formulations and numerical methods are needed in order to overcome the computational limitations arising from the fastest waves in the atmosphere.
Accurately modeling atmospheric phenomena at horizontal resolutions below
10 km necessitates moving to a non-hydrostatic formulation of the governing
equations. The step size constraints from sound waves can be addressed either
by removing the fast waves in the model with a soundproof formulation of the
equations or using a numerical method that can stably step over the fastest
waves. The latter approach includes split-explicit
Split-explicit methods typically divide the dynamics into three groups: fast
vertical waves that are treated implicitly, fast horizontal waves that are
substepped relative to the other dynamics with an explicit method, and slow
dynamics that are updated with an explicit method using a long time step
With the push toward exascale computing, there has been increasing interest
in evaluating the potential of IMEX methods for efficiently simulating
atmospheric dynamics at high resolution.
In this work, we investigate the performance of 21 Runge–Kutta IMEX
methods from the literature, including many of those tested in
The choices of IMEX partitioning, integration method, and implicit solver are
evaluated in terms of accuracy and efficiency using the Tempest
non-hydrostatic dynamical core
In the following section, we present the formulation of non-hydrostatic
equations implemented in Tempest, followed by a discussion of the spatial and
temporal discretizations and splitting approaches in
Sect.
The non-hydrostatic dry-atmosphere/shallow-atmosphere equations in the
Tempest
The non-hydrostatic equations are discretized using a method of lines
approach. First, the terms on the right-hand sides of
Eqs. (
The spatial discretization of Eqs. (
In Tempest, hyperviscosity is employed in the horizontal directions by
default. The operators are fourth-order derivatives with nominal coefficients
of
There are numerous approaches for integrating the system of ordinary differentia equations (ODEs) resulting
from the spatial discretization of
Eqs. (
The spatially discretized non-hydrostatic equations can be written as a
general initial value problem with the right-hand side additively split into
two parts:
A particular ARK method is defined by a combination of an explicit and a
diagonally implicit pair of Butcher tableaux:
While Eqs. (
We investigate a number of ARK methods from the literature with a variety of
numerical properties:
classical second- (ARS232, ARS222, and ARS233) and third-order (ARS343 and ARS443)
methods from the third- (ARK324), fourth- (ARK436), and fifth-order (ARK548) methods
from the second-order ARK232 method derived for the NUMA model and presented
in second- (SSP2(222), SSP2(332)a, and SSP3(332)) and third-order (SSP3(433))
strong stability preserving methods from second-order strong stability preserving method SSP2(332)b from
third-order strong stability preserving methods SSP3(333)b and SSP3(333)c
from
The ARK232, ARS232, ARS233, ARS443, SSP2(222), SSP2(332)a, SSP3(332), and
SSP3(433) methods were previously examined by
With the exception of ARS233 and SSP3(333)a, b, and c, all of the above
methods are constructed with an L stable implicit method. Thus, the implicit
portion of the method is accurate in the limit of the stiff term becoming
infinitely fast, meaning that slow dynamics are resolved while fast modes,
e.g., acoustic waves, are damped. Two methods, ARS233 and SSP2(222), are
B stable which is a nonlinear stability indicating that the difference
between two numerical solutions does not increase with time. Several methods
are SSP and are designed to maintain the total
variation diminishing (TVD) property of a spatial discretization. The
optimized SSP methods from
In this section, we present four HEVI formulations of the non-hydrostatic equations
(Eqs.
Based on the principles above, terms in
Eqs. ( HEVI-A, where all vertical dynamics except vertical advection of horizontal velocity
in Eqs. ( HEVI-B, where vertical velocity advection in Eq. ( HEVI-C, where thermodynamic advection in Eq. ( HEVI-D, where vertical velocity advection in Eq. (
In addition to the HEVI options, we also consider IMEX splittings that solve
various parts of horizontal dynamics implicitly. These formulations contain
the same vertically implicit terms as HEVI-A but add some of the horizontal
terms into the implicit function:
IMEX-A, where the density equation (Eq. IMEX-B, where the density (Eq.
The IMEX-A option treats the density equation with a single consistent
scheme, while IMEX-B is motivated by semi-implicit splittings
An
Newton's method finds the solution of Eq. (
Newton's method can be quite expensive, especially when many iterations are
needed to achieve convergence, since each iteration involves computing or
approximating the Jacobian matrix and performing a linear solve. As an
alternative, we also consider treating Eq. (
In both solver approaches, the solution value
Finding the solution of the nonlinear system
(Eq.
The inclusion of horizontal dynamics in the implicit function introduces
coupling between degrees of freedom located in different columns, and a
linear solve over the full domain is necessary to compute the Newton update.
In this case, we employ a Newton–Krylov approach for the nonlinear solve
where an approximate solution of Eq. (
Accuracy
We evaluate the accuracy and computational efficiency of the various
implicit–explicit splittings, ARK methods, and solver options on two test
cases. Section
The gravity wave test as defined in
Accuracy and efficiency plots are shown in
Figs.
Accuracy
Accuracy
Accuracy
Accuracy
Accuracy
The second-order ARK methods can be divided into two groups based on accuracy
regardless of the splitting choice. The more accurate group of methods
consists of the lpm1, lpm2, lpum, and lspum optimized variants of SSP2(332)
from
The approximate largest stable step size is consistent across the HEVI
splittings. The ARK232, ARS232, and SSP3(332) methods are stable with
The relative efficiency of the different ARK methods is also consistent
across the splitting options. Despite requiring three implicit solves per
time step, the optimized SSP2(332) methods from
Across the splitting options, the majority of the third-order ARK methods
produce solutions with approximately the same level of accuracy, with the
exception of SSP3(433), which is generally more accurate, and ARS233,
SSP3(333)b, and SSP3(333)c, which are less accurate. The fourth-order accurate
ARK436 has smaller errors than all second- and third-order methods, and the
fifth-order ARK548 method generally has the lowest error overall. The
fifth-order ARK548 does not achieve the expected convergence rate, and with the
IMEX-A splitting all of the methods drop to second-order convergence. Since
the IMEX-B and HEVI-A methods show no such deterioration in accuracy, and
they match IMEX-A but have more/fewer implicit terms, respectively, we
believe that IMEX-A suffers from order reduction in the coupling terms.
Specifically, it is likely that IMEX-A splits two large and opposite terms
into explicit and implicit components, whereas IMEX-B and HEVI-A treat both
terms consistently. As a result, partial derivatives of
Approximate largest stable step size (in seconds) for a 30-day run
of the baroclinic wave test. Second-order methods are shown in the top
section of the table and higher-order methods in the bottom section. For
entries separated by “/”, the left value is the step size for the
Rosenbrock-like approach and the right value is the step size for the Newton
solver. When a single step size is given, the Rosenbrock-like and Newton
solvers were stable at the same step size. As in the gravity wave tests, the
choice of a Rosenbrock-like or Newton solver does not generally impact the
maximum stable step size, except in the case of IMEX-B which fails to
converge. While the methods are able to complete a 30-day run at the step
sizes listed below, the solutions produced are not sufficiently accurate in
all cases and depend on the solver choice.
Table
Like the second-order methods, the choice of HEVI splitting does not effect the approximate maximum step size of a given third-order method. ARS233, ARS443, and SSP3(333)a, b, and c all have a maximum steps size of 1 s, and ARS343, SSP3(433), ARK324, ARK436, and ARK548 allow steps of up to 2 s. SSP3(333)a is the only method to show a doubling in the maximum step size, going from 1 to 2 s, due to the additional implicitness in IMEX-A. In the IMEX-B tests, all of the methods, with the exception of SSP3(333)a which does not gain stability, have an increase in maximum step size to 8 s. As with second-order methods, the IMEX-B splitting is the only option where the choice of a Rosenbrock-like approach alters the integrator results by reducing the maximum step size due to lack of solver convergence.
Among the third-order methods, SSP3(443) is the most efficient method, except at the smallest step sizes where convergence begins to slow, and SSP3(333)a becomes faster for the same accuracy. Likewise, the fourth- and fifth-order methods are more cost effective until the convergence slows at the smallest time step sizes. SSP3(333)a is the best approach for lower accuracy levels in IMEX-B and is the best scheme in IMEX-A. For higher accuracy with IMEX-B, the faster convergence of ARK436 and ARK548 makes these approaches more efficient until convergence begins to slow at small step sizes. As with the second-order methods, HEVI-B, C, and D do not present an advantage over HEVI-A in run time, and the additional communication required by the horizontal terms in the implicit portion of the IMEX methods is not offset by sufficient gains in step size.
With both the second- and higher-order integration methods, HEVI-A with the Rosenbrock-like approach is the best combination in this test case. For the most part, third-order methods outperform the second-order methods in terms of accuracy at a given step size. Since the third-order methods do not increase the maximum stable step size over that achieved by the second-order methods, the second-order schemes are more efficient at looser error requirements, and higher-order methods are best when more accuracy is necessary.
The maximum vertical velocity with ARS343 using the Rosenbrock-like approach (solid lines) and the Newton solver (dashed line) for various time step sizes. The light red region defines the 99 % confidence interval from the explicit simulations with perturbed initial conditions, and the light purple region is 10 % of the maximum deviation in the 99 % confidence interval.
The second test case simulates the development of a baroclinic wave over the
course of approximately 10 days as described in
Since this problem produces a strong instability, comparisons against a
highly resolved reference solution, as was used in the gravity wave test, do
not yield a good metric for quality of a numerical solution. To define an
acceptable numerical solution generated by the methods at any given time
step, the results of the implicit–explicit simulations (HEVI or IMEX) are
compared against the range of maximum vertical velocities produced by five
explicit simulations with initial conditions perturbed by random noise. For a
state variable
Approximate largest step size (in seconds) for a 30-day run of the baroclinic wave test that produces acceptable maximum vertical velocities over time. Second-order methods are shown in the top section of the table and higher-order methods in the bottom section. For entries separated by “/”, the left value is the step size for the Rosenbrock-like approach and the right value is the step size for the Newton solver. When a single step size is given, the Rosenbrock-like and Newton solvers gave acceptable solutions at the same step size.
The corresponding run times for the approximate largest acceptable
step sizes in Table
Unlike the gravity wave test, treating the thermodynamic advection explicitly (HEVI-C and D) reduces the maximum stable and acceptable step size for some of the integration schemes. As a result, the increased number of time steps with HEVI-C and D can lead to longer run times than with HEVI-A or B depending on the ARK method. Treating only the vertical velocity advection explicitly (HEVI-B) does not impact the maximum stable or acceptable step size, nor does it offer a significant advantage in run time over the HEVI-A setup. Handling more terms implicitly in IMEX-A and B can greatly increase the maximum stable step size but, in general, this does not translate into faster run times due to the increased solve cost and the smaller step sizes need to produce a sufficiently accurate solution. However, in a few cases with the Rosenbrock-like approach (ARK232, ARS222, ARS232, SSP2(332)lpm1, and SSP2(332)lpum), IMEX-A runs are faster than results with HEVI-C and D because of the larger acceptable time step size with the IMEX-A splitting and the minimal increase in solver cost due to the effectiveness of the vertical solve as a preconditioner (only two to four linear iterations are required per Newton iteration). The preconditioner is less effective in the IMEX-B splitting as more dynamics are included that are not treated by the preconditioner, so 16 to 25 linear iterations are needed per Newton iteration. As in the gravity wave test, the Newton solver does not perform well with the IMEX-B splitting and is unable to converge at step sizes for which the Rosenbrock-like approach gives an acceptable answer.
In this test, the increased accuracy and larger stability regions of the higher-order methods enable bigger time step sizes than the second-order methods with HEVI splittings and are somewhat less affected by the choice of HEVI splitting. The gains in step size are large enough to offset the third implicit stage solve required for ARK324 and ARS343, which consistently perform well. The ARS343 method is the fastest method across the HEVI splittings. ARS233, SSP3(333)b, and SSP3(333)c are less robust to the choice of splitting and solver but, when they produce an acceptable solution (HEVI-A and B with the Newton solver), are the second fastest methods, as they only require two implicit solves per step and have relatively large acceptable step size. ARS324 is more robust to the choice of spitting and solver, and is the third fastest method with HEVI-A and B, and the second fastest method with HEVI-C and D. The second-order ARK232 and ARS232 methods give nearly identical performance and tie for fourth fastest method with the HEVI-A and B splittings.
The ARK324 and ARS343 methods also highlight the potential advantage of the Newton solver over the Rosenbrock-like approach. With the exception of the second-order SSP methods (discussed below), the second-order methods studied produce acceptable solutions at their largest stable step size for HEVI splittings with either a Newton or Rosenbrock-like approach. As a result, there is no benefit from using the Newton solver for second-order methods and the Rosenbrock-like approach is always more efficient. At the larger stable step sizes enabled by higher-order methods, a Rosenbrock-like approach does not always give a sufficiently accurate answer, and a smaller step size is necessary to produce a good solution. Iterating to a converged stage value leads to better results at larger step sizes and, since only a few nonlinear iterations are necessary (on average two iterations per stage solve), a HEVI splitting with a Newton solver can outperform the Rosenbrock-like approach when the step size gain is sufficiently large.
While the other higher-order schemes are also able to take larger time steps than the second-order methods, they require more function evaluations or implicit solves than ARK324 or ARS343, and the step size gains are not enough to overcome the additional costs. Four of the third-order methods (ARS233, SSP3(333)b, and SSP3(333)c with the Rosenbrock-like solver, and SSP3(333)a with either solver) are not stable for 30 days at step sizes of at least 100 s with any of the splittings. These failures are likely because the implicit parts are not L stable (or even A stable for SSP3(333)a), and the fastest dynamics of the system are not sufficiently damped. These methods did perform well in the gravity wave test case, which might have been due to the reduced domain size altering the eigenvalues of the system.
However, L stability does not guarantee that a method will produce a good
solution. All of the SSP methods tested, with the exception of SSP3(333)a, b,
and c, are L stable but only SSP2(332)a consistently gives acceptable results
with the HEVI splittings. The other SSP methods are generally stable but give
vertical velocities an order of magnitude larger than the mean solution value
with step sizes above 100 s. Exceptions to this behavior include SSP2(332)b,
which underestimates the vertical velocities, and SSP3(333)b and c, which have
acceptable solutions at their maximum stable step size when using the Newton
solver with the HEVI-A or B splittings. The better performance of SSP3(333)b
and c may be attributable to having the same
Considering the results of the gravity wave and baroclinic wave test cases, the HEVI-A and B approaches are the most accurate and efficient of the implicit–explicit splittings considered. Treating some of the vertical dynamics of HEVI-A explicitly does not provide a noticeable gain in efficiency from simpler implicit systems and, in the case of HEVI-C and D, can lead to reduced step sizes in the baroclinic wave test. Adding horizontally implicit terms to the HEVI-A formulation does increase the maximum stable step size, but the gains are not large enough to overcome the added cost of a globally implicit solve.
While SSP methods are the most accurate and efficient approaches in the gravity wave test case, they generally do not preform well in the baroclinic wave test (with some notable exceptions), possibly due to error from the choice of implicit–explicit splitting. The reduced domain size seemed to skew the gravity wave test results in favor of these methods while the other ARK schemes perform well in both test cases. Additionally, the gravity wave test case does not show a benefit, in terms of maximum stable step size, with higher-order methods, although it does highlight their greater efficiency when higher accuracy is required. Again, these results are likely due to the reduced domain size altering the eigenvalues of the system. In the baroclinic wave test on a full-size Earth, higher-order methods produce accurate solutions at step sizes large enough to have faster run times than second-order schemes involving fewer implicit solves.
At the larger time step sizes enabled by higher-order methods in the baroclinic wave tests, the choice of nonlinear solver approach becomes an important consideration. A Rosenbrock-like approach limits the cost associated with multiple Newton iterations but may require a reduced step size to obtain an accurate solution. Taking larger steps is possible by iterating stage solutions to convergence with Newton's method. The additional cost is minimal and can be offset by the larger step size. The choice of predictor values was not considered in this work but could lead to more efficient nonlinear solves with Newton's method or more accurate Rosenbrock-like schemes.
The HEVI-A and B configurations produced nearly identical results, while the
HEVI-C and D options were problematic for some methods in the baroclinic wave
test. Since HEVI-A employs the same discretization for kinetic energy
transport as vertical mass transport without a significant difference in
computational cost, it is preferred over the HEVI-B option. Overall, the
third-order ARS343 method shows excellent performance across the splitting
and solver options. ARS233, SSP3(333)b, and SSP3(333)c are also efficient
third-order methods but their performance depends on the appropriate choice
of splitting and solver. A more robust runner-up method is the third-order
ARK324 method which follows closely behind ARS343 in run times. The
second-order ARS232 and ARK232 methods highlighted in
The ARK324 and ARK232 are of particular interest as both include an embedded method which will be leveraged for future studies on temporal adaptivity in atmospheric simulations using ARKode. Varying the time step size can enable greater efficiency by placing temporal accuracy where is it needed most to capture dynamics of interest. Additionally, we plan on further evaluating the methods in this study on the 2016 dynamical core model intercomparison project (DCMIP2016) test cases to better understand the impacts of coupling with simplified physics on performance of implicit–explicit splittings and integration methods.
Tempest is available through the Git repository at
In Table number of implicit solves per step ( number of explicit stages ( order – theoretical order of accuracy of the explicit Runge–Kutta (ERK) method, the diagonally implicit Runge–Kutta (DIRK)
method, and the overall ARK method (including coupling conditions); stage order – theoretical order of accuracy of stages (relevant for
order reduction on stiff problems), again for the ERK stages, DIRK stages, and the overall ARK
stages; stability – A, L, and B stability for the DIRK portion of each method;
Properties for each of the ARK methods used in this paper. The column headings are described in the above text.
S.A. DIRK – if the DIRK method is stiffly accurate (i.e., the last
row of S.A. ERK – if the ERK method is “stiffly accurate” (i.e., the
last row of same solution weights ( same abscissa ( maximum stable explicit step along the imaginary axis – as this
application has purely imaginary eigenvalues, we numerically compute the
largest
The authors declare that they have no conflict of interest.
Support for this work was provided by the Department of Energy, Office of Science Scientific Discovery through Advanced Computing (SciDAC) project “A Non-hydrostatic Variable Resolution Atmospheric Model in ACME”. This work was performed under the auspices of the US Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. LLNL-JRNL-737448. Edited by: Simone Marras Reviewed by: two anonymous referees