A new anelastic large-eddy simulation (LES) model with an Eulerian dynamical core and Lagrangian particle-based microphysics is presented. The dynamical core uses the multidimensional positive-definite advection transport algorithm (MPDATA) advection scheme and the generalized conjugate residual pressure solver, whereas the microphysics scheme is based on the super-droplet method. Algorithms for coupling of Lagrangian microphysics with Eulerian dynamics are presented, including spatial and temporal discretizations and a condensation substepping algorithm. The model is free of numerical diffusion in the droplet size spectrum. Activation of droplets is modeled explicitly, making the model less sensitive to local supersaturation maxima than models in which activation is parameterized. Simulations of a drizzling marine stratocumulus give results in agreement with other LES models. It is shown that in the super-droplet method a relatively low number of computational particles is sufficient to obtain correct averaged properties of a cloud, but condensation and collision–coalescence have to be modeled with a time step of the order of 0.1 s. Such short time steps are achieved by substepping, as the model time step is typically around 1 s. Simulations with and without an explicit subgrid-scale turbulence model are compared. Effects of modeling subgrid-scale motion of super-droplets are investigated. The model achieves high computational performance by using graphics processing unit (GPU) accelerators.

Over the last decade, Lagrangian particle-based cloud microphysics schemes have been drawing increasing attention.
They are similar to Eulerian bin schemes in that they explicitly model the size spectrum of droplets and explicitly resolve microphysical processes, but have a number of advantages over them

In this paper, we document the development of a new large-eddy simulation (LES) model called the University of Warsaw Lagrangian Cloud Model (UWLCM).
It is an anelastic model with a finite-difference Eulerian dynamical core and a Lagrangian microphysics scheme.
The Lipps–Hemler anelastic approximation

One of the key reasons for developing a new model is to use a modern software development approach. The model code is written in the C++ programming language and makes use of many mature libraries available in that language (e.g., Blitz++, Boost and Thrust). The code is open-source and under a version-control system. A set of automated tests greatly helps in ensuring the correctness of the model. The automated tests include a 2-D moist thermal simulation, a 2-D kinematic stratocumulus simulation and a test of different combinations of model options. Moreover, modeling of physical processes, e.g., condensation, advection, coalescence, sedimentation, is tested separately by the libmpdata++ and libcloudph++ test suites. UWLCM makes efficient use of modern computers that have both central processing units (CPUs) and graphics processing units (GPUs). The Eulerian computations of the dynamical core are carried out on CPUs and, simultaneously, Lagrangian microphysical computations are undertaken on GPUs. However, it is also possible to run the Lagrangian microphysics on CPUs.

Some results obtained using earlier versions of UWLCM have already been published.
In

Section

Eulerian prognostic variables of the model are the potential temperature

The set of anelastic Lipps–Hemler equations

UWLCM offers two methods for modeling diffusion of Eulerian variables due to the SGS turbulence.
The first is an implicit LES (ILES) approach, in which there is no explicit parametrization of SGS mixing, i.e.,

Liquid water is modeled with a Lagrangian, particle-based microphysics scheme from the libcloudph++ library

The condensational growth rate of a SD is equal to that of a single real particle.
We calculate it using the Maxwell–Mason approximation (see

Collision–coalescence of droplets is treated as a stochastic process

Such treatment of coalescence, sometimes referred to as the “all-or-nothing algorithm”, assures that coalescence does not increase the number of SDs.
This algorithm was found to give the best results in a recent comparison of various coalescence algorithms used in Lagrangian schemes for microphysics

Super-droplets are treated as non-inertial particles that always sediment with their terminal velocity.
There is an option to model the diffusion of liquid water due to the SGS turbulence by adding a random velocity component

Numerical integration of the governing Eulerian equations is carried out using the MPDATA algorithm implemented in libmpdata++

The pressure perturbation

UML sequence diagram showing the order of operations within a single time step. Calls in boldface start microphysical calculations that are carried out on GPUs simultaneously with solver operations undertaken on CPUs. The RHS is divided into condensational and non-condensational parts,

For numerical reasons, condensational growth of SDs is solved in terms of the squared wet radius

The stochastic collision–coalescence process described in Sect.

The procedure for the initialization of SD sizes is described in detail in

Advection of SDs is modeled with a predictor–corrector algorithm described in

The sequence of operations undertaken in a single time step is presented on a Unified Modeling Language (UML) sequence diagram in Fig.

Eulerian dependent variables of the model are co-located.
Their positions form the nodes of the primary grid.
However, the libmpdata++ advection algorithms are formulated using a dual, staggered Arakawa-C grid

Schematic of a 2-D computational domain. Bullets mark the data points for the
dependent variable

UWLCM simulations of a marine stratocumulus cloud are presented in this section.
The main goal is to validate UWLCM by comparing it with other LES models.
We study sensitivity of the model to the way the SGS turbulence is modeled and to values of the microphysics scheme parameters.
Results of this sensitivity study may provide some guidance to other users of Lagrangian microphysical schemes.
The simulation setup is based on observations made during the second Dynamics and Chemistry of Marine Stratocumulus (DYCOMS-II) field study

The simulation setup follows

The 2-D simulations are used to investigate sensitivity of the model to parameters of the SDM: the coalescence time step length

The 2-D UWLCM results.
Time series of the domain averaged liquid water path, entrainment rate (equal to

Time series of selected domain averaged variables are shown in Fig.

Vertical profiles from the 2-D simulations are shown in Fig.

A conclusion for SDM modeling is that coalescence needs to be resolved with a time step of the order of 0.1 s, although more rigorous convergence tests should be carried out in the future.
This conclusion is surprising, because coalescence tests of SDM in box models give correct results for time steps larger than 1 s (result not shown), and

The 2-D UWLCM results.
As in Fig.

As in Fig.

Based on conclusions of the 2-D sensitivity test, 3-D simulations are undertaken for

Profiles from 3-D simulations are presented in Fig.

As in Fig.

The purpose of this section is to study discrepancy between the amount of surface precipitation observed during the DYCOMS-II campaign (from ca.

Selected time series of domain averages (

Time series and profiles showing the amount of liquid water, surface precipitation and concentration of cloud droplets from UWLCM, DHARMA and RAMS are plotted in Fig.

In the bin microphysics of RAMS and DHARMA, water droplets are artificially divided into haze particles and cloud droplets, and droplet activation is modeled as an instantaneous process

We presented the University of Warsaw Lagrangian Cloud Model (UWLCM), a new large-eddy simulations model with Lagrangian particle-based cloud microphysics. The model is built by combining two open-source libraries, one for handling Eulerian dynamics and the other for implementing the Lagrangian microphysics scheme. Methods for coupling Lagrangian microphysics with Eulerian dynamics were presented, including spatial discretization, substepping algorithms and an algorithm for simultaneous computations of Eulerian and Lagrangian components. Simulations of a marine stratocumulus show that the model produces results that are in agreement with reference results from 11 other LES models, which proves the capability of Lagrangian microphysics to model realistic clouds. Both 2-D and 3-D simulations of the stratocumulus were performed. The 2-D simulations with UWLCM give reasonable results regarding microphysical phenomena at a fraction of the computational cost of the 3-D simulations, and were used to study sensitivity of the Lagrangian microphysics scheme. It was found that the condensational and collisional growth of droplets has to be modeled with a 0.1 s time step and that the number of computational particles does not affect domain averages, apart from small changes in the precipitation flux. The 0.1 s time step for condensation and coalescence is realized by carrying out multiple time steps for these processes at each model time step. Different approaches to modeling SGS turbulence were compared in 3-D simulations. The best agreement with other models is obtained using the Smagorinsky scheme and an algorithm for the SGS turbulent motion of computational particles. The implicit LES approach is troubled by the lack of diffusion of liquid water represented by Lagrangian computational particles. Surface precipitation modeled by UWLCM is lower than observed. This suggests that some physical phenomena not modeled by UWLCM, such as SGS turbulence affecting condensation and coalescence of droplets or giant CCN, are important for precipitation formation. In UWLCM, all particles, including humidified aerosols, evolve according to the same set of equations. Therefore, it is not necessary to include droplet activation as an additional process. Advantages of such an approach are most apparent near the cloud base, where bin schemes produce local maxima of cloud droplet concentration, whereas in UWLCM cloud droplet concentration increases monotonously.

The UWLCM, libmpdata++ and libcloudph++ source codes are available at

For simulation results, please contact Piotr Dziekan.

List of symbols.
As in

Consider condensation of SDs within cell

We expect the difference between the two substepping algorithms to be particularly large near a moving cloud edge.
We test this hypothesis by simulating cloud edge advection in an idealized 1-D setup, considering two spatial cells: one at saturation (

Errors caused by substepping in the 1-D simulation of cloud edge advection.
The error is defined as

The per-cell algorithm causes artificial evaporation of droplets, and this error increases with the number of substeps, whereas the per-particle algorithm produces correct results independently of the number of substeps. Therefore, we conclude that in simulations in which there is a lot of cloud edge advection it is necessary to use the per-particle algorithm.

However, in simulations with little cloud edge advection, the per-cell algorithm might be sufficient.
We check this by using different substepping algorithms in a 2-D marine stratocumulus simulation (the DYCOMS RF02 case, cf. Sect.

Vertical profiles of the concentration of cloud droplets from the 2-D piggybacking simulations with different substepping algorithms, averaged over the 2–4 h period.
The black line and shaded regions show results from

UWLCM is coded in the C++ language.
It relies heavily on two C++ libraries developed by the cloud modeling group at the University of Warsaw: libmpdata++

libmpdata++ is a set of solvers for the generalized transport equations that use the MPDATA advection scheme. The solvers are organized in a hierarchy, ordered from solvers for simple flows to solvers for more complex flows. Each more complex solver inherits from the simpler solver in the hierarchy. Such a design simplifies code development, maintenance and reusability. Apart from the hierarchy of solvers, libmpdata++ contains three other independent modules: boundary conditions, concurrency handlers and output handlers, all of which are used in UWLCM.

libcloudph++ is an implementation of three microphysical models: SDM, a single-moment bulk model and a double-moment bulk model. The SDM is implemented using the Thrust library and the CUDA programming language. Therefore, the SDM can be run on multi-threaded CPUs as well as on multiple GPUs.

The UWLCM code is built on top of the libmpdata++ solvers.
Separate parts of the UWLCM code handle different types of simulations.
The “piggybacking” code makes it possible to run kinematic simulations, i.e., simulations with a prescribed velocity field.
The “2-D/3-D” part of the code handles the dimensionality of the problem.
The “forcings” code specifies external forcings, so it is the part of the code that depends on the simulation setup.
The “microphysics” module is responsible for handling the choice of microphysics (only Lagrangian microphysics is available in the current UWLCM release).
Owing to the code structure, different types of simulations, e.g., 2-D and 3-D simulations, different simulation setups or kinematic simulations, mostly use the same source code.
The highest performance is achieved when UWLCM is run on a system with GPUs.
In this case, the Eulerian component is calculated on CPUs and the Lagrangian component on GPUs.
A large part of these computations is carried out simultaneously (cf. Fig.

Schematic depiction of the structure of the code of UWLCM, libmpdata++ and libcloudph++. Black arrows denote inheritance between classes.

PD developed the model code with contributions from MW. PD performed the simulations. PD and HP prepared the paper with contributions from MW.

The authors declare that they have no conflict of interest.

We thank Wojciech Grabowski for discussions about the model formulation and the paper. We are grateful to the Academic Computer Center CYFRONET AGH for providing computing power.

This research has been supported by the Polish National Science Center (grant no. 2016/23/B/ST10/00690).

This paper was edited by Klaus Gierens and reviewed by two anonymous referees.