Weather radars are ideally suited for providing the input data for precipitation nowcasting at high resolution, namely spatial scales under 2 km and time ranges between 5 min and 3 h . Despite recent advances in numerical weather prediction e.g.,, extrapolation-based nowcasting remains the primary approach at such space- and timescales, typically outperforming NWP forecasts in the first 2–5 h, depending on the weather situation, domain and NWP characteristics e.g.,. Other recent developments include machine learning methods, for which promising results have been obtained e.g.,, but these have not so far been deployed in operational nowcasting systems.

Precipitation exhibits variability over a wide range of space- and timescales e.g., which, in combination with the chaotic nature of the atmosphere e.g.,, limits our ability to predict its evolution in a deterministic manner. The NWP community recognized this challenge in the early 1990s and tackled the problem by producing an ensemble of NWP forecasts by perturbing the set of initial conditions e.g.,. Those perturbations grow exponentially and lead to an ensemble of solutions that reflect forecast uncertainties. The information contained in the ensemble can then be used to derive probabilistic forecasts.

Just as any other forecasting technique, the skill of radar-based nowcasting was found to depend on multiple factors such as the meteorological conditions, geographical location, spatial and temporal scales e.g.,. It is therefore not surprising that also the nowcasting community rapidly acknowledged the importance of estimating predictive uncertainty e.g.,. A common approach is based on stochastic simulation, in which correlated noise fields are used to perturb a deterministic nowcast e.g.,. Substantial research efforts have been made to make the perturbation fields as realistic as possible and consistent with the nowcast uncertainty e.g.,. For a review of the history of nowcasting starting from the 1950s, and its evolution to the probabilistic framework, we refer the reader to .

Similarly to other research fields, the nowcasting community has invested a significant amount of time to re-implement from scratch routines and algorithms that have been around for decades, for example, optical flow and advection schemes. Part of this problem is due to the unavailability of software, which is often proprietary or too poorly documented to be understood, trusted and used.

Recognizing that nowcasting methods and related applications can be further developed and distributed by promoting universal access to existing knowledge, a Python-based software package, called pysteps, is being developed as a community-driven effort. Such effort fits well into the weather radar community with emergence of open data and an increasing number of open-source software projects , for instance, in radar data processing . More recently, community-based initiatives dedicated to nowcasting have emerged, for example, Com-SWIRLS by the Regional Specialized Meteorological Centre (RSMC) for Nowcasting operated by the Hong Kong Observatory (HKO), IMPROVER by the UK MetOffice or rainymotion at the University of Potsdam (see Table ).

In this article, we present pysteps, an open-source and community-driven Python library for probabilistic precipitation nowcasting. The objective of pysteps is two-fold. First, it aims at providing a well-documented and modular framework for development of new nowcasting methods. In this sense, pysteps promotes the adoption of open-science practices, as the lack of common standards, transparency, code availability and well-documented workflows in computational disciplines can lead to non-reproducible results, hence questioning their scientific value . Second, pysteps aims at providing an easily accessible software package for practitioners ranging from weather forecasters to hydrologists.

The paper is structured as follows. The theoretical framework for precipitation nowcasting and using stochastic perturbations to characterize the uncertainty is formulated in Sect. . The general architecture of the pysteps library is presented in Sect. . A comprehensive verification of pysteps nowcasts is given in Sect. . Various experiments to understand the sensitivity of pysteps to the model parameters and define the default configuration are done in Sect. . The limits of pysteps are tested in Sect.  using a tropical cyclone and severe convection case in Australia. Section  concludes the paper and lists potential future applications of pysteps. Finally, code listings demonstrating the use of pysteps are given in Appendix .

In its simplest form, extrapolation-based precipitation nowcasting assumes that over the time frame of a few hours the evolution of precipitation can be captured by moving the radar echoes along a stationary motion field without changes in intensity. In the literature, this is known as Lagrangian persistence .

Denoting a precipitation parcel by R and its displacement vector by α(τ), the conservation equation for an incompressible flow is written as 1R(x0;t+τ)=R(x0-α(τ);t), or equivalently in differential form as 2dRdt=∂R∂t+u∂R∂x+v∂R∂y,u=dxdt,v=dydt, where dR/dt=0, and u and v are the x and y components of the motion field. In the so-called optical flow methods, u and v are estimated for a given location by solving Eq. () numerically based on a sequence of precipitation intensity fields. Typically, a constraint on the spatial continuity of nearby u and v is imposed to guarantee a unique solution. Once the motion field is known, the radar echoes are extrapolated by means of an advection scheme.

Three methods are currently implemented in pysteps for motion field estimation: a local Lucas–Kanade method , a global variational echo-tracking approach and a spectral approach DARTS, . The currently implemented advection method is the backward-in-time semi-Lagrangian scheme described in , which is robust against numerical diffusion.

The predictability of the atmosphere is intrinsically limited by the fact that its state cannot be observed with absolute precision nor expressed without approximations in its governing laws . In the case of radar-based precipitation nowcasting, predictive uncertainty originates from errors in the estimation of the current state of the rainfall and motion fields (initial state errors), and limitations of Lagrangian persistence as a model to predict the evolution of the rainfall and motion fields (model errors).

The main contribution to model errors in the Lagrangian approach stems from the evolution of precipitation in terms of initiation, growth, decay and termination processes that violate the steady-state assumption. Other sources of model uncertainty include the assumption of stationarity of the motion field, inaccuracies due to the practical implementation of the method, as the discretization in time, space and reflectivity, and numerical diffusion of the advection scheme .

Currently, pysteps focuses on the representation of the model errors, whereas incorporation of the initial state errors in the nowcasting is left for future work.

The statistics of intermittent precipitation rates are non-Gaussian and display a typical asymmetric distribution that is bounded at zero. Such properties restrict the usage of well-established stochastic models that assume Gaussianity. A common workaround is to introduce a suitable data transformation to approximate a normal distribution e.g.,.

Currently, pysteps assumes a log-normal distribution of rain rates by applying the logarithmic transformation 3R→10log⁡10R,ifR≥0.1mmh-1-15,otherwise, that corresponds to logarithmic radar rain rates (units of dBR). The value of -15 dBR is equivalent to assigning a rain rate of approximately 0.03 mm h-1 to the zeros. Hereafter, R refers to the transformed rain rates, unless otherwise stated.

Using the logarithmic transformation is motivated by the fact that rain rates are approximately log-normally distributed . This has two main advantages. First, it simplifies the estimation of distribution parameters, particularly with limited sample size and in the presence of measurement noise . Second, the decomposition of log-transformed rainfall fields defines a multiplicative cascade, where multiplication is replaced with summation in the transformed space .

It has been shown that the lifetime of precipitation relates to its spatial scale e.g.,, often denominated as dynamic scaling. Recognizing this fundamental property, introduced the Spectral Prognosis (S-PROG) model, which laid the foundation for the development of Short-Term Ensemble Prediction System (STEPS) . The key idea is to decompose the precipitation field into a multiplicative cascade, where the cascade levels represent different spatial scales, and treat them separately in the nowcasting model.

In STEPS, the scale decomposition is done by applying a fast Fourier transform (FFT) to the input precipitation field. This is motivated by the fact that for a grid of size L×L pixels, the radial Fourier wavenumbers |k|=kx2+ky2 are related to spatial scales via 4|k|︷radialwavenumber(pixels)→L|k|︷wavelength(pixels)→LΔx|k|︷wavelength(km)→LΔx2|k|︷scale(km), where Δx denotes the grid resolution (km). Thus, the spatial scale is half the wavelength. Alternative approaches to perform a scale decomposition include the discrete-cosine transform or wavelets .

In the current implementation of pysteps, we adopt the approach of , where Gaussian weight functions are used for separating the Fourier spectrum into a set of radial bands. An example of the weight functions for the domain covered by the Finnish Meteorological Institute (FMI) radars is shown in Fig. . After the FFT and Gaussian filtering, each frequency band is transformed back to the spatial domain, which results in a cascade with n levels each representing a different scale (see an example in Fig. ).

In nowcasting, the typical approach to model the temporal evolution of precipitation fields employs an auto-regressive (AR) process that combines the deterministic component from Lagrangian persistence with a stochastic innovation term, also referred to as a noise or perturbation term. For instance, S-PROG and STEPS use a second-order AR(2) process with two parameters. Separate AR(2) processes are applied to each cascade level to account for the dynamic scaling of precipitation. The combination of the auto-regressive model in time and the cascade model in space allows one to control the temporal evolution and correlation structure of precipitation.

Currently, a more general AR(p) model has been implemented in pysteps. For each cascade level j, the recursion formula is given by 5Rj(x,y,t)=∑k=1pϕj,kRj(x,y,t-kΔt)+ϕj,0εj(x,y,t). The first term corresponds to the deterministic predictable component at cascade level j (i.e., Lagrangian persistence). The second term is a stochastic term that represents the unpredictable component at the same cascade level j, that is, mainly initiation, growth and decay of precipitation. The symbol Δt denotes the time difference between consecutive precipitation fields Rj that are normalized to zero mean and unit variance.

The parameters ϕj,k in the above model are estimated from time-lagged auto-correlation coefficients ρj,k for k=1,2,…,p using the Yule–Walker equations . For p=2, the correlation coefficients can be adjusted to ensure that the resulting AR(p) process is stationary and non-periodic . Finally, the parameters ϕj,0 are chosen as 6ϕj,0=1-∑k=1pρj,kϕj,k. Given that the variance of the noise fields εj is one, this choice guarantees that the AR(p) process is normalized to unit variance .

The theoretical auto-correlation function (ACF) of the AR(2) process can be computed recursively from the model parameters and auto-correlation coefficients according to 7ρj(t)=ϕj,1ρj(t-Δt)+ϕj,2ρj(t-2Δt). The empirical ACF can be derived by computing the correlation coefficients between the extrapolation nowcasts and the observations.

For an exponentially decaying ACF, the precipitation lifetime is defined as the time when the ACF, theoretical or empirical, falls below the value 1/e≈0.37, where e is the Euler number. Alternatively, one can estimate the lifetime by integrating the ACF according to 8T=∫0∞ρ(τ)dτ.

It is not common to employ an AR(p) process with p>2 for several reasons. First, it is not trivial to guarantee the stationarity and non-periodicity of the process. Second, when estimated in Lagrangian frame, the higher-order auto-correlation coefficients are affected by the uncertainty of the motion field. This occurs especially at small spatial scales as it is difficult to properly track convective cells over several time steps. Third, a low-order AR process is generally sufficient to model the loss of predictability in the nowcasting range; departures are usually observed only after ≈2 h .

The perturbation field ε in Eq. () is typically generated as a correlated Gaussian random field using FFT filtering e.g.,. The process consists of three steps:

generate a Gaussian white noise field,

A second source of uncertainty in Lagrangian persistence nowcasting stems from temporal evolution of the motion field . This can be accounted for by adding stochastic perturbations. In the current implementation of pysteps, this is done by applying the method of .

For simplicity, the perturbation field is assumed to be spatially constant for each ensemble member, but the magnitude of the perturbations increases with respect to lead time. For a given initial advection field w0 and lead time t, the perturbed velocities are given by 12wp(x,y)=w0(x,y)+αpar(t)εpar(x,y)w^par+αperp(t)εperp(x,y)w^perp, where w^par and w^perp denote the components parallel and perpendicular to the initial advection field w0, respectively. The random variables (εpar and εperp) are sampled from the Laplace distribution with zero mean and unit variance. Scaling of the perturbations is done according to 13αpar(t)=apartbpar+cpar14αperp(t)=aperptbperp+cperp, where the parameters are climatologically fitted by using a large sample of advection fields. Example values of these parameters can be found in Table .

To ensure that the forecast fields have the same statistical properties with the observed ones, post-processing is typically done at the very end of the chain. This is necessary because intermediate steps may introduce discrepancies. One major source of such discrepancies is related to the difficulty to model the intermittency of precipitation. Typically, the basic statistical properties such as wet-area ratio, mean, variance and the marginal distribution of precipitation intensities are assumed to remain invariant through the nowcast.

In the present implementation of pysteps, the post-processing involves two different types of methods: (1) masking and (2) matching the statistics of the forecast fields with the most recently observed ones. Methods of type (1) are used to avoid generation of stochastic cells into areas that are too distant from existing precipitation. Methods of type (2) can be applied unconditionally or only to pixels within the mask.

Three different masking methods have been implemented. In the first method, the mask is obtained from pixels exceeding an intensity threshold in the observed precipitation field, and the mask is kept constant in Lagrangian coordinates for the whole forecast. In the second method, adapted from , the mask is obtained by using the S-PROG (i.e., the unperturbed STEPS) nowcast. In the third method, a lead-time-dependent precipitation mask is applied. The mask is defined by the pixels exceeding a given intensity threshold in the observed precipitation field and then progressively relaxed to allow the stochastic evolution of the wet area.

Two methods have been implemented for matching the statistics of forecast fields with the observed ones. In the first method, which is used together with the S-PROG mask, the conditional mean of the masked forecast field is adjusted to match the conditional mean of the observed field (excluding intensities below the threshold). Alternatively, the cumulative distribution function (CDF) of the forecast field can be mapped to the observed one . This is defined as 15R′(x,y)=Fobs-1(F(R(x,y))), where Fobs and F denote the CDFs of the observed and the input forecast field R, respectively.

The implementation language of pysteps is Python (http://python.org, last access: 23 September 2019). As a high-level language with an extensive built-in standard library and a large number of external libraries available, it is ideally suited for open-source software development. Python distributions, such as Anaconda, providing the necessary software to run pysteps are available for all major platforms. Python also provides interfaces for compiled languages such as C/C++ and Fortran, allowing to improve performance in time-critical modules. In addition, Python-based tools, like the IPython shell or the Jupyter notebooks (https://jupyter.org, last access: 23 September 2019), allow an interactive use of pysteps for research and demonstration purposes.

The pysteps library is extensively documented. The documentation describes in detail the different modules and the application programming interfaces (APIs). The modules are documented by using the docstring concept of Python. This is implemented using Read the docs (https://readthedocs.org/, last access: 23 September 2019) and Sphinx (http://www.sphinx-doc.org/en/master, last access: 23 September 2019) to automatically compile and update an online version of the documentation, available at https://pysteps.readthedocs.io (last access: 23 September 2019). In addition, tutorials for performing various tasks with pysteps are included as example scripts.

Pysteps development is done by using git, a distributed version control system. The source code of pysteps is hosted in GitHub (https://pysteps.github.io, last access: 23 September 2019). In addition to code hosting, the features of GitHub include development in multiple branches, issue tracking and wiki pages. Developers outside the core team may fork the main repository and integrate the proposed changes via pull requests, which allow community-driven development. Continuous integration and testing is done by using the Travis CI framework (https://travis-ci.com/pySTEPS/pysteps, last access: 23 September 2019).

Pysteps is published under the three-clause BSD license. It allows copying, redistribution and modification of the software as long as the modification are tracked and the source code is made available under the same license. The permissive license model makes the software easily accessible to potential users, even allowing use for commercial purposes.

Pysteps relies on several external libraries that are listed in Table . It is built on top of NumPy, SciPy and Matplotlib, that together provide a MATLAB-like computing environment in Python. These libraries provide data structures and wrappers for low-level BLAS and LAPACK libraries for high-performance matrix and array operations, image processing methods and also high-level functionality for data visualization. The NumPy array is the basic data structure used in pysteps.

Support for NetCDF (the default file format), HDF5 and various image file formats is implemented via the NetCDF4, h5py and PIL libraries. A complete list of supported input/output file formats is given in the official pysteps documentation. Plotting precipitation data with basemaps has been implemented via mpl_toolkits.basemap and cartopy packages. The Lucas–Kanade optical flow algorithm used in pysteps is implemented in the OpenCV library and accessed via a Python interface. Parallelized computation of nowcast ensembles is done by using Dask, which provides a platform-independent back end for low-level methods.

The aim of pysteps is to be a modular software library where all the main components are interchangeable. This makes the pysteps an ideal research platform for developing and testing new methods as well as a valuable tool for operational meteorology, easily allowing the comparison of different nowcast algorithms or running multi-model ensemble nowcasts. Pysteps is currently divided into 11 modules that perform different tasks. The modules and their descriptions are listed in Table .

The modularity is implemented via interface-based design. To this end, each module implements one subtask and an interface method for retrieving the desired method for this task. All mutually interchangeable methods implement the same interface. Another key principle is that whenever possible, the data are stored in n-dimensional arrays, which allow an efficient and compact representation.

The above design principles are demonstrated in the following example. A precipitation nowcast by using STEPS can be generated by >> nowcast_method = nowcasts.get_method("steps") >>> nowcast = nowcast_method(R, V, num_timesteps)]]> where the required inputs are  

R: array of shape (t,m,n) containing a time series of t observed precipitation intensity fields with shape (m,n),

Additional parameters can be specified by using keyword arguments. The output of stochastic nowcasting methods is a four-dimensional array of shape (num_ensemble_members, num_timesteps, height, width). For deterministic nowcasts, the first dimension is dropped.

In addition to being modular, pysteps implements object-oriented features. However, instead of using customized classes, we use dictionaries and functions that operate on the dictionaries similarly to class member functions. This design decision is motivated by the principle of using the core Python data types rather than implementing customized classes. The flat design of pysteps should facilitate user interaction and embedding of individual modules and functions in other software. In this way, pysteps is similar to wradlib .

To demonstrate the above design, the following example shows how to construct a Gaussian bandpass filter for eight cascade levels using the filter_gaussian function implemented in the cascade.bandpass_filters module: >> filter = filter_gaussian(R.shape, 8)]]> The output is a dictionary with three elements: (1) one-dimensional weights corresponding to the radial wavenumbers, (2) a two-dimensional weight field for the FFT of the input image and (3) a list of central frequencies for each weight function (see Fig. ). The resulting filter object can then be passed to decomposition_fft as follows: >> decomp = decomposition_fft(R, filter)]]> The decomposition is applied to a two-dimensional precipitation field R, and the output is again a dictionary with three-elements: (1) a three-dimensional array containing the eight cascade levels having the same dimension as R, (2) mean precipitation values of each cascade level and (3) standard deviations for each level. More detailed examples of pysteps usage are provided in Appendix .

Figure  illustrates the workflow for generating precipitation nowcasts using pysteps. The first step is reading the input data using the io module. Methods for reading radar composites from Australian Bureau of Meteorology (BoM), FMI, KNMI and MeteoSwiss have been implemented. In addition, the importers module supports reading the European-wide OPERA radar composites in the ODIM HDF5 format. Conversion from reflectivity (dBZ) to precipitation intensity (mm h-1) and other preprocessing can be done by using the utils module. Due to the modular design of pysteps, reading custom data formats and conversions (e.g., by using different Z-R relationships and polarimetric parameters) can also be implemented.

Reading the input data is followed by determination of the motion field using the methods implemented in the motion module. The precipitation intensity and motion fields are supplied as inputs to a user-chosen nowcasting method implemented in the nowcasts module. For the Lagrangian persistence method implemented in nowcasts.extrapolation, the remaining step of generating the nowcast is extrapolation.

When the cascade decomposition and the autoregressive model are used for scale filtering (the S-PROG model), the additional steps include those marked with green color in Fig. . When generating ensembles (the STEPS model), stochastic perturbations are added to the AR(p) models and to the advection field using the methods implemented in the noise module. These steps are marked with blue color in Fig. .

The ensemble generation is parallelized by using the dask library. For each time step, this is done by splitting the computation to the available processor cores so that each core is responsible for computation of one ensemble member.

Given the input radar composites and the motion field, all operations involved in generating a nowcast are called from the nowcast module, except optional post-processing. This can be done either by supplying the requested method to the nowcast generator or separately by using the functionality implemented in the postprocessing module. The post-processing includes methods to ensure that the nowcasts have the same statistical properties of the observations (see Sect. ), as well as methods for generating different products, such as ensemble mean or exceedance probabilities of given intensity thresholds. Computation of accumulations from instantaneous rain rates can be done by using the methods implemented in the utils.dimension module. Finally, the nowcasts or nowcast ensembles can be verified and plotted by using the verification and visualization modules, respectively.

As of 2019, the radar network operated by the FMI consists of 10 polarimetric C-band Doppler radars. After clutter filtering, the measured radar reflectivities are interpolated into a grid with spatial and temporal resolutions of 1 km and 5 min, respectively. The correction for the vertical profile of reflectivity (VPR) is applied in order to reduce range-dependent biases . Finally, reflectivities are converted to rainfall intensities using the Z-R relation, Z=223R1.53, adapted to the Finnish climate conditions . A total of 10 precipitation events from Finland containing both stratiform and convective precipitation were chosen for this study (Table ).

The latest fourth-generation MeteoSwiss network consists of five polarimetric C-band Doppler radars . The quantitative precipitation estimation (QPE) product used in this study includes automatic hardware calibration, clutter filtering, correction for beam shielding, correction for VPR effects, Z-R relation (Z=316R1.5) and bias adjustment . The radar composite is calculated on a 1 km grid every 5 min. Overall, 10 events consisting of predominantly convective precipitation were chosen from the Swiss data (Table ).

The US dataset comprises the radar mosaics provided by Warning Decision Support System-Integrated Information WDSS-II, covering the continental United States at a spatial resolution of approximately 1 km. For the WDSS data, the resolution of the precipitation fields is upscaled from 1 to 4 km by averaging 4×4 grid points to reduce the computational requirements. The chosen precipitation events are described in Table .

The radar network operated by BoM consists of 66 radars, mostly C-band Doppler radars, with S-band polarimetric Doppler radars operating at four major cities. Raw reflectivity observations are quality controlled in real time to remove non-meteorological echoes and estimate the reflectivity at the Earth's surface. This equivalent reflectivity at the surface is converted into an instantaneous rainfall rate by use of power–law functions tuned on a per radar basis. Finally, rainfall depths are estimated by adjusting the bias of instantaneous rainfall rates based on observations at real-time gauge locations. The QPE grids are calculated with a spatial resolution of 0.5 km every 5 min. The BoM radar dataset comprises two precipitation events: a tropical cyclone in northern Australia and a severe convective event in Sydney (Table ).

Table  summarizes the different data sources and resolutions.

Pysteps includes a number of verification metrics to help users to analyze the general characteristics of the nowcasts in terms of consistency and quality (or goodness). Probabilistic forecasts have been verified using the relative operating characteristic (ROC) curve, reliability diagrams and rank histograms, as implemented in the verification module of pysteps.

The ROC curve measures the ability of a probabilistic forecast to discriminate between precipitation and no precipitation exceeding a given intensity threshold. For a set of probability thresholds, the ROC curve is constructed by plotting the probability of detection (POD) against the false alarm rate (POFD), not to be confused with the false alarm ratio (FAR). For a perfect forecast, the curve passes through the upper left corner (i.e., 100 % hit rate and 0 % false alarm rate). The area under the ROC curve can be used as a measure of potential skill. For more details on the contingency tables and the formulas of the categorical scores, the reader is referred to .

The reliability diagram measures the bias (reliability) and resolution of a probabilistic forecast. For a given intensity threshold, the diagram shows the forecast probability against the observed frequencies, where the probability range [0,1] is divided into n bins. For a perfectly reliable forecast, the curve lies on the diagonal. The reliability diagram is often accompanied by a histogram showing the sample size in each bin (sharpness diagram). A sharp forecast has few samples in the middle of the histogram and many on the sides (probability of either 1 or 0).

The rank histogram measures how well the ensemble spread corresponds to the observed uncertainty. For each nowcast grid pixel, the ensemble members are ranked in increasing order. A pooled histogram is computed by assigning each verifying observation a bin which it falls into among the ensemble members. The first and last bins are assigned for observations below or above all members, respectively. For a forecast ensemble whose distribution is consistent with the observations, the histogram is flat and no observations fall into the first or last bin. To handle ties (e.g., when both the observed precipitation and several ensemble members are equal to 0), we implemented the method of . The method randomly chooses a bin between (M+1) and (M+Mtied)+1, where M is the number of members smaller than the observation and Mtied is the number of ties (ensemble members equal to the observation).

An additional metric that can be derived from rank histograms is the outlier percentage (OP). The OP measures the proportion of observations falling outside the ensemble, defined by 16OP=h1+hn+1∑i=1n+1hi, where hi denotes the ith bin of the rank histogram.

Pysteps also includes standard neighborhood verification methods, such as the fractions skill score (FSS). FSS provides an intuitive assessment of the dependency of skill on spatial scale from high-resolution precipitation forecasts . The FSS is computed by comparing the forecast and observed fractional coverage of precipitation exceeding certain thresholds in spatial windows (neighborhoods) of increasing size. Using FSS it is possible to determine how the forecast skill varies with neighborhood size and then determine the smallest scale that provides a sufficiently skillful forecast.

The quality of ensemble nowcasts produced by pysteps was verified by using the MeteoSwiss data and the default configuration listed in Table . Using the reliability diagram, ROC curve and rank histogram as verification metrics, the results of the experiments are shown in Figs. –. The results obtained by using the FMI data were very similar and thus not shown here.

Figure  shows that for the 0.1 mm h-1 intensity threshold, reliable and sharp nowcasts can be obtained up to 2 h. The ROC area remains over 0.85, and the deviation of the reliability diagrams from the diagonal remains below 0.25. However, there is a noticeable loss of sharpness after 3 h. In addition, the curved shape of the reliability diagrams indicates that the pysteps nowcasts are slightly overconfident .

When a higher 5 mm h-1 intensity threshold is used, Fig. a shows a significant deviation of the reliability diagram from the diagonal only after 45 min, which is accompanied by loss of sharpness. However, the ROC area remains above 0.8, indicating potentially useful skill. This suggests that more reliable nowcasts could be obtained by implementing additional calibration procedures in a future version of pysteps. Another observation that suggests lack of calibration is that the optimal nowcasts for precipitation/no precipitation are obtained by choosing a very low probability threshold (for a well-calibrated nowcast, this would be 0.5).

The rank histograms (Fig. ) also show some ensemble underdispersion with larger values on the first and last bins. In general, we found that there are more misses than false alarms (i.e., cases when all members are lower than the observations). This occurs, for instance, in cases of convective initiation. Despite the ability of pysteps to generate some new light random rain, it is not designed to represent the uncertainty related to an explosive initiation of a thunderstorm.

Conventional semi-Lagrangian schemes are implemented in a recursive way so that the precipitation intensities are interpolated at each time step, which usually leads to substantial numerical diffusion (i.e., loss of power at high spatial frequencies). In the pysteps method (the extrapolation.semilagrangian module), this is done by iteratively tracing the locations of precipitation parcels and interpolating the intensities only as the final step of the advection .

To verify the advantage of this implementation, we computed radially averaged Fourier spectra of deterministic nowcasts at various lead times for FMI event no. 3 (Fig. ). The analysis is performed using the three optical flow methods to understand whether the semi-Lagrangian scheme is sensitive to quality of the motion field. Figure  shows an almost perfect overlap of the forecast and observed spectra, an indication that the numerical diffusion of the semi-Lagrangian scheme is very low. Several cases have been analyzed and provided similar results (not shown).

The aim of probabilistic nowcasting is to generate a reliable ensemble of equiprobable realizations of precipitation fields characterized by power spectra similar to those in Fig. . Figure a shows the average spectra of a stochastic 48-member nowcast for FMI precipitation event no. 3. Despite a small loss of power for scales (<100 km), all spectra are close to the observations. In other words, the spatial structure of ensemble members remains realistic at all forecast lead times.

Figure b shows the results of the same analysis but for the ensemble mean forecast (the average of ensemble members). The process of ensemble averaging should produce precipitation fields that become smoother with lead time, which is the aftermath of the loss of predictability at small scales . As expected, Fig. b shows a gradual loss of power at small scales. The departure of the forecast spectra from the observed ones occurs at increasing wavelengths, i.e., ≈16 km at 5 min and ≈128 km at 60 min. However, after 30 min, there is a certain increase of power at wavelengths smaller than 16–32 km. This behavior is attributed to the limited ensemble size, which is not large enough to filter out precipitation features at small scales. Thus, one may argue that if the ensemble is too small to model the loss of predictability at such scales, it may also be too small to reliably model the forecast uncertainty.

An alternative way to deterministically represent the forecast uncertainty is to filter out the unpredictable features using the S-PROG model (Fig. c). Also, in this case, the departures of forecast spectra from the observed one occur gradually as in the ensemble mean. The first two lead times are remarkably similar, while for lead times beyond 30 min the S-PROG filtering is stronger (at small spatial wavelengths). Again, this level of filtering could be reached with an ensemble of infinite size.

The previous result suggests that we could exploit the discrepancies between the S-PROG and ensemble mean spectra to obtain an estimate of the required ensemble size (as a function of spatial scale and lead time). If the two spectra are similar, it is an indication that the ensemble is large enough.

To demonstrate the effectiveness of the hierarchy of AR(2) models in modeling the temporal evolution of precipitation, we derived the theoretical ACF from the estimated AR parameters (see Eq. ). The obtained ACF is compared to the empirical ACF between the nowcasts and the corresponding observations. The correlation coefficients are computed separately for each cascade level obtained using the bandpass filters shown in Fig. .

Figure  shows the average theoretical and empirical ACFs for all the FMI cases. It clearly indicates that the AR(2) process gives accurate estimates of the temporal auto-correlations up to 3 h. For smaller scales (0–35 km) having short lifetimes, the estimates coincide nearly exactly with the observed ones, but for larger scales the auto-correlations are slightly overestimated. This is due to the relatively short memory of the AR(2) process compared to the precipitation lifetimes at these scales (over 2 h).

Determination of the advection field by optical flow is a key component of any extrapolation-based nowcasting system. Pysteps allows to easily analyze the impact of the optical flow method and also scale filtering on the forecast skill. Moreover, the three methods currently available in the motion module constitute an ideal testbed as they cover three very distinct approaches; see the references in Sect.  for details. The experiments were done by using the MeteoSwiss and US precipitation events described in Tables  and .

Each optical flow method was used with two deterministic nowcasting methods: a simple extrapolation-based method and S-PROG, which incorporates a scale filtering procedure as described in . Both methods are available in the nowcasts.extrapolation and nowcasts.sprog modules, respectively. The VET and Lucas–Kanade methods use two input images, while DARTS uses nine input images. The forecast quality was evaluated using the critical success index (CSI) and the mean absolute error (MAE) as described in .

The results of the experiments are shown in Fig. . First of all, large differences between the simple extrapolation and S-PROG nowcasts are observed, which is mainly due to the scale filtering implemented in S-PROG (see Sect. ). For the MeteoSwiss events, applying the filtering improves both CSI and MAE, especially at longer lead times (Fig. a and c). After 2 h, the S-PROG nowcasts show a ∼20 % increase in the CSI and ∼40 % reduction in the MAE. A similar behavior is observed for the US events (Fig. b and d) but with a ∼20 % the reduction in MAE after 2 h.

On the other hand, no significant differences can be observed between different optical flow methods (less than 2 %), with DARTS performing slightly worse than the other methods. This is possibly explained by the fact that, with the default configuration, DARTS produces only a large-scale approximation of the advection field.

Figure  shows advection fields obtained using different optical flow methods for a selected case (US, 11 April 2013 at 08:00 UTC). Lucas–Kanade and VET produce smooth fields that are remarkably similar, particularly close to the precipitation areas (Fig. a and b). Within precipitation areas, DARTS produces similar motion fields to the other two methods, but outside precipitation the fields are considerably different.

We also measured the computation times of different optical flow methods in the MeteoSwiss and FMI domains, and the results are shown in Table . The experiments were done using an Intel Xeon E5645 CPU with 12 cores running at 2.4 GHz with parallelization enabled in the optical flow methods. The results reflect the fact that the Fourier space and local methods (DARTS and Lucas–Kanade) have significantly lower computational requirements than variational methods (VET), which are however still within the needs of a real-time operational system. Thus, our conclusion from the results shown in Fig.  and Table  is that the choice of the optical flow method plays a less significant role, while nowcast errors are more clearly determined by the dynamic scaling properties of precipitation as highlighted by the large impact of scale filtering on the forecast skill.

The ensemble size is one of the main factors contributing to the quality and computation time of pysteps nowcasts, and one has to make a trade-off between these two. To determine the optimal number, the skill of the nowcasts with different intensity thresholds and ensemble sizes was evaluated by using two metrics. These are the area under the ROC curve and the outlier percentage (OP). The results are shown in Figs.  and .

Figure  shows that the choice of the ensemble size plays a significant role, which is particularly true when nowcasts of higher precipitation intensities are desired. Figure a shows that for n=6, the ROC area falls below 0.85 after 2 h, while it is close to 0.9 when n is increased to 48. However, there is only marginal improvement when n is increased from 24 to 48, which suggests that 24 members are sufficient when nowcasts of precipitation/no precipitation are desired with low-intensity thresholds (e.g., 0.1 mm h-1). On the other hand, Fig. b shows that when the threshold is increased to 5 mm h-1, a significant improvement can be expected when increasing n from 48 to 96 or even over 100.

The OP is highly dependent on the ensemble size, which can be observed from Fig. . With 96 ensemble members, OP is below 15 % after 20 min, which indicates that the ensembles are well able to capture the uncertainties in the spatiotemporal evolution of precipitation. The OP could be further reduced by increasing the ensemble size over 100. Another observation from Fig.  is the significant dependence of OP on the lead time. The highest OP can be observed at 20 min, and after 3 h it is up to 50 % smaller.

We also analyzed the computation times needed to generate nowcast ensembles. In a real-time setting, it is essential to know how many ensemble members can be produced before the arrival of the next input radar rainfall image (usually every 5 min). To this end, 1 h nowcasts were computed with different ensemble sizes and number of parallel threads using the FMI and MeteoSwiss data listed in Tables  and , respectively.

The results of the above experiments are shown in Fig. . Figure a shows that for the input grid of 710×640 pixels used in the MeteoSwiss domain, it is possible to generate 1 h nowcast ensembles of up to 48 members in less than 2 min using a server with 12 processor cores.

The results for the larger FMI domain with grid size of 760×1226 pixels are shown in Fig. b. Compared to the MeteoSwiss domain, the height of the grid is doubled, which also doubles the computation time (the computational complexity increases quadratically with respect to grid size). Nevertheless, using 12 processor cores, the computation time of a 48-member ensemble still remains below 2 min.

In addition, Fig. a and b show the effectiveness of the parallelization scheme implemented in pysteps. That is, when plotted in logarithmic scale, the computation time decreases approximately linearly with respect to the number of threads (i.e., the computation time is halved when the number of threads is multiplied by 2).

This experiment investigates the impact of localization on the nowcast quality. In this context, localization means restricting the nowcasting model into small subdomains instead of applying it the whole domain assuming spatial homogeneity of the precipitation field, as in the earlier STEPS implementations e.g.,. To this end, the short-space approach presented in for stochastic noise generation is generalized to the whole nowcasting system (see module nowcasts.sseps). Adapting the approach described in , the parameter estimation and the nowcasting model are implemented in a moving window of predetermined size. The localization is applied to the cascade decomposition, the autoregressive process (), the non-parametric Fourier filter () and the probability matching ().

The impact of localization is assessed in terms of rank histograms and reliability diagrams (threshold of 1.0 mm h-1) for a 30 min lead time (Fig. ). The localization shows positive effects in the ensemble spread, which improves both in terms of reliability and conditional bias, although we also observe a slight decrease of sharpness. This is reflected in the rank histograms, which tend to get flatter as the localization window gets smaller. This seems to be mainly driven by a reduction in the proportion of observations lying above the ensemble, which reduces from approximately 13 % to 8 %.

The cascade decomposition was designed to account for dynamic scaling (i.e., the dependence of predictability on spatial scale; see Sect. ). Without the decomposition, precipitation fields are expected to evolve similarly at all spatial scales following a single AR process. In such cases, the lifetime of small-scale (large-scale) precipitation features would be overestimated (underestimated). Thus, our main hypothesis is that dynamic scaling properties are necessary to produce a realistic temporal evolution (lifetime) of precipitation across spatial scales. Consequently, this would give correct ensemble dispersion because the standard deviation of the perturbations is inversely related to predictability via Eq. ().

To test our hypothesis, we compared the stochastic nowcasts (nowcasts.steps module) with and without cascade decomposition, that is, using eight or one cascade levels, respectively. The objective is to analyze the realism of the temporal evolution, not whether the AR is an appropriate model of the forecast error as in Sect. . In practice, this implies comparing the theoretical ACFs of forecast and observed fields as follows: (1) generate nowcasts with either an eight-level or one-level cascade, (2) transform the forecast fields into the Lagrangian frame (by using the same motion field estimated at start time), (3) decompose the forecast fields into a six-level cascade, (4) estimate the AR(2) parameters at each scale, (5) derive the full temporal ACF (see also Fig. ) and (6) integrate the ACF to estimate the precipitation lifetime. The procedure is repeated for each forecast lead time up to 2 h and also for the corresponding observations.

Figure  shows the average lifetime for all the MeteoSwiss events plotted against spatial wavelength (in log–log scale). As expected, the model with eight cascade levels reproduces well the dynamic scaling properties, especially at small wavelengths. However, there is some degree of overestimation of the lifetime at large wavelengths compared to the observations. One possibility would be to adjust the AR parameters to obtain faster decorrelation, and thus a shorter lifetime, at such scales.

The model without cascade decomposition compensates for the overestimation of persistence at large wavelengths but strongly overestimates the one of small wavelengths. Hence, the evolution of convective cells in the stochastic nowcast is too slow compared with reality. This could be checked visually by looking at the animations of stochastic realizations with and without decomposition (*_stoch_*.gif in https://github.com/pySTEPS/pysteps-publication/tree/master/animations, last access: 23 September 2019).

Another approach to understand the impact of the cascade decomposition is to analyze the filtering properties of the ensemble mean forecast e.g.,. Figure  illustrates the evolution of the ensemble mean forecast spectra with eight and one cascade levels, respectively. When using the cascade decomposition the process of ensemble averaging leads to a loss of power at small spatial wavelengths, in agreement with the expected loss of predictability (see Sect. ). Instead, the model with one cascade level is not able to filter out the unpredictable features. As a consequence, it may not be able to adequately characterize the loss of predictability (and uncertainty) at different spatial scales.

Figure  illustrates the ensemble and probabilistic verification for all the MeteoSwiss events with and without cascade decomposition. The sensitivity of forecast uncertainty estimations on using the incremental precipitation mask is also included.

The rank histograms behave differently depending on the chosen forecast settings (Fig. a). The two models without decomposition denote a clear overdispersion with a characteristic dome-shape in the bin range 13–22, especially for the setting with one level and no mask. Instead, the models with eight levels display a flat histogram, except for the very last bin, which contains the frequency of observations exceeding all the ensemble members (misses). As shown in Fig. , this underdispersion can be reduced by increasing the ensemble size. The last bin is also quite sensitive to using the mask, which prevents the ensemble to capture the uncertainty associated to precipitation initiation far from the main precipitation body.

Figure b shows the spread–error relationship analysis (i.e., the standard deviation among all ensemble members) against the average RMSE of all members against the observations. The experiments with eight levels have both a lower RMSE and spread than the ones using one level. It can also be noticed that the one-level models do not show the same overdispersion that was observed on the rank histograms.

Finally, the reliability diagrams of Fig. c–d demonstrate a very good reliability for all forecast settings, although the forecast probabilities of the models with one level are slightly lower than the observed frequencies. In addition, the eight-level model has better sharpness, i.e., a larger proportion of high forecast probabilities (>0.9).

Figures  and show examples of FSS results calculated by pysteps for different forecast times for both Australian case studies.

The FSS decays in both case studies when spatial scale is reduced or when the intensity threshold is increased, although differences exist between the two case studies. For example, the tropical cyclone case seems to have a less acute reduction in the skill with changes in spatial scale. This can be related to the presence of a more uniform rainfall distribution across the domain (large bands of rainfall moving in an organized way) limiting displacement errors at small scales. Instead, the skill reduces heavily as rainfall intensity increases. This drop in skill could have been accentuated by the relatively low number of high-intensity samples in these events.

On the other hand, the severe convection case displays a stronger decay of skill when spatial scale is reduced, probably due to the presence of sharp spatial gradients and isolated convective cells. This said, it is interesting to note how for the higher intensities and large spatial scales the FSS values do not decay as heavily as seen in the other case study. This difference could be a consequence of having more high-intensity values in the severe convection event.

To compare the behavior of the AR(2) model for these two case studies, temporal auto-correlation functions for each spatial scale were calculated using Eq. () and then integrated to estimate the precipitation lifetimes for each scale and runtime. Figure  summarizes the precipitation lifetime results for each case study. Overall, a more diverse set of spatial and temporal patterns observed during the severe convection event makes interquartile ranges of precipitation lifetimes larger for this case study for all scales. In comparison, similar organized patterns were present during most of the duration of the tropical cyclone event and therefore precipitation lifetime values have a narrower range. Smaller scales seem to have similar average lifetime values for both cases with no strong temporal variations within the events. For the larger scales, however, precipitation lifetime values for tropical cyclone event are greater than severe convection ones, again as a consequence of large-scale organized patterns observed in this event.

From an operational perspective, these results illustrate the importance of using an AR(2) model with parameters continuously adjusted to the latest observed patterns to adequately simulate rainfall nowcasts instead of using fixed, historical parameters. However, it is important to note as well that a number of outliers were obtained in both cases (mainly for the larger spatial scales). These anomalous values may indicate the need for introducing a temporal smoothing scheme during the estimation of the AR(2) parameters. Having a more stable, slowly evolving parameters would help to (i) reduce the possibility of generating unrealistic nowcasts from one particular set of observations and also (ii) create smooth transitions between consecutive rainfall nowcasts.

Pysteps represents a long-term effort that does not end with the publication of this paper. The current pysteps version (1.0) provides a quite comprehensive library but still misses two important modules: (1) a module to generate QPE ensembles characterizing the radar measurement uncertainty e.g., and (2) a module for seamless blending of precipitation fields from different data sources, such as radar nowcasts and NWP forecasts , radar, satellite and NWP data .

It would be interesting to include other state-of-the-art ensemble precipitation nowcasting systems in pysteps, for example, PHAST , SBMcast , SAMPO-TBM , SWIRLS ensemble rainstorm nowcast SERN; and NowPrecip . A large-scale forecast verification intercomparison project could be foreseen to better understand the advantages and disadvantages of different ensemble nowcasting techniques.

Pysteps opens a number of possibilities that go beyond the field of nowcasting. The most natural application of pysteps is to use the precipitation ensembles as inputs into hydrological models for uncertainty quantification, in both urban and rural environments e.g.,.

An obvious and crucial application of nowcasting systems is to support the operational warnings for rainstorms, thunderstorms and severe weather.

Individual pysteps modules can also serve different purposes. For example, the optical flow modules can be used to study precipitation growth and decay in moving coordinates e.g.,, to correct radar field accumulations accounting for advection e.g.,, to synchronize the individual radar elevation scans e.g., or to separate the location error of NWP precipitation forecasts .

The tools available in the noise and the time series modules can be used for stochastic simulation of design storms e.g., and weather generators , and also to understand and quantify the subpixel variability of radar rainfall e.g.,. Other applications can include stochastic downscaling or emulation of climate model output e.g.,.

We encourage the nowcasting community and potential users to implement new nowcasting methods, propose new modules, try pysteps on different applications, provide feedback and contribute to the library for the benefit of everyone.