The first step is to build a database of analogs of the atmospheric circulation. We outline the procedure of , summarized in Fig. a. For a given day t, we determine the similarity of Z500 for all days t′ that are within 30 calendar days of t but in a different year from t. The similarity is quantified by a Euclidean distance (or root mean square error) between the daily Z500 maps. Other types of similarity measures are possible , but the expected impact on the results is often marginal . We believe that the simplicity of the Euclidean distance makes it more robust to changes in horizontal resolution (e.g., from NCEP to ERA5), compared with more sophisticated distances that include local spatial gradients, which would require adjustments and additional tuning. This choice can be left open for future fine-tuning, depending on the region.

For each day t, we consider the K best analogs, i.e., for which the distances are the smallest. We compute the spatial rank correlation between the Z500 best analogs and the Z500 at time t for posterior verification purposes.

As a refinement over the study of , a time embedding of τ days was used for the search of the analogs dates. This means that the field X(t) for which we compute analogs is X(t)=(Z500(t),Z500(t+1),…,Z500(t+τ)). This ensures that temporal derivatives of the atmospheric field are preserved . Hence, the distance that is optimized to find analogs of the Z500(x,t) field is 1D(t,t′)=∑x∑i=0τZ500(x,t+i)-Z500(x,t′+i)212, where x is a spatial index and τ is the embedding time.

We consider different geographic domains as shown in Fig.  for the computation of analogs and weather regimes. The computation of circulation analogs was performed with the “blackswan” Web Processing Service (WPS; ). The “blackswan” WPS is an online tool that helps compute circulation analogs on various datasets (e.g., reanalyses and climate model simulations) with a user-friendly interface.

We use a stochastic weather generator (SWG) based on a random sampling of the circulation analogs. The operation of the SWG and its design are detailed by . The aim is to generate random trajectories from the previously computed analogs. Therefore, to generate a trajectory, we proceed as follows: for a given day t0 in year y0, we generate a set of N=100 simulations until a time t0+T, with a lead time T∈{5,10,20} d. We start at day t0 and randomly select an analog (out of K analogs) of day t0+1. The random selection of analogs of the day t0+1 is performed with weights that are proportional to the calendar difference between t0 and analog dates, to ensure that time goes forward. We also exclude analog dates with years that are equal to y0. This rule is important for the next iterations. We then replace t0 by the selected analog of t0+1 and repeat the operation T times. Excluding analogs in year y0 from the selection ensures that we do not use information from the T days that follow t0. Hence, we obtain a hindcast trajectory between t0 and t0+T.

The procedure presented above is repeated N=100 times to simulate N=100 trajectories from t0 to t0+T0. The daily precipitation of each trajectory is time averaged between t0 and t0+T. Hence, we obtain an ensemble of N=100 forecasts of the average precipitation for day t0 and lead time T.

Then t0 is shifted by Δt≥1 d, and the ensemble simulation procedure is repeated. This provides a set of ensemble forecasts with analogs.

We made a hindcast exercise, where the forecasts of precipitations based on analogs of atmospheric circulation (Z500), are started every Δt≈ T/2 d between 1 January 1948 and 31 December 2019. This yields a stochastic ensemble hindcast of precipitation and atmospheric circulation (Z500). In this paper, therefore, we analyze the properties of an ensemble forecast of mean precipitation between t0 and t0+T. To evaluate our forecasts, the predictions made with the SWG are compared with the persistence and climatological forecasts. The persistence forecast consists of using the average value between t0-T and t0 for a given year. The climatological forecast takes the climatological mean between t0 and t0+T. The two “reference” forecasts are randomized by adding a small Gaussian noise, whose standard deviation is estimated by bootstrapping over T long intervals. We thus generate sets of persistence forecasts and climatological forecasts that are consistent with the observations .

The simulations of this stochastic model will be called “SWG forecasts”, as opposed to ECMWF forecasts.

Forecast verification is the process of determining the statistical quality of forecasts. A wide variety of ensemble forecast verification procedures exists ​​​​​​​. They involve measures of the relationship between a set of forecasts and corresponding observations. To assess the quality of precipitation forecasts, we compute indicators such as the correlation and continuous rank probability skill score (CRPSS) for each lead time T, for different seasons and months.

The temporal rank correlation (referred to as correlation skill) is calculated between the precipitation observations and the median of 100 simulations. This simple diagnostic is often used to assess forecast skills of indices .

The continuous ranked probability score (CRPS) is widely used for probabilistic forecast verification . It is sensitive to the distance between forecast and observation probability distributions.

If the ensemble forecast x yields a probability distribution P(x) for a value xa, the CRPS measures how the probability distribution of x compares with xa .

The CRPS is computed as 2CRPS(P,xa)=∫-∞+∞P(x)-H(x-xa)2dx, where xa is the observation and H is the Heaviside function of the occurrence of xa (H(y)=1 if y≥0, and H(y)=0 otherwise). The decomposition and properties of the CRPS have been investigated by , , and . A perfect forecast would have a CRPS equal to 0, but the CRPS value obviously depends on the units of the variable to forecast, so quantifying what is a “good” forecast requires a normalization. It is hence difficult to compare CRPS values for temperature and precipitation, within the same ensemble forecast. This issue is also acute for non-Gaussian variables with heavy tails so that the interpretation of a given CRPS value might not be informative.

One way of circumventing this difficulty is to compare CRPS values to reference forecasts, such as persistence or climatology. The continuous rank probability skill score (CRPSS) is a normalization of Eq. () with respect to such a reference.

The CRPSS is hence computed by 3CRPSS=1-CRPS‾CRPSref‾, where CRPS‾ is the time average of the CRPS of the SWG forecast and CRPSref‾ is the time average of the CRPS of the reference (either climatology or persistence). The CRPSS is interpreted as a fraction of improvement over a reference forecast.

The values of the CRPSS vary between -∞ and 1. The forecast is considered to be an improvement over the reference when the CRPSS value is positive. Values of CRPSS equal to 0 indicate no improvement over the reference. Values inferior to 0 mean that the forecast is worse than the reference.

We use the CRPSS values to determine the maximum lead time T for which the SWG forecast is better than references. Then the SWG assessments will use the CRPS and directly compare the probability distributions of precipitation ensemble forecasts.

We investigated the role of North Atlantic weather patterns on the forecast quality by attributing CRPS values of the SWG precipitation simulations to weather regimes. Weather regimes are defined as large-scale quasi-stationary atmospheric states. They are characterized by their recurrence, persistence, and stationarity . They help in describing the features of the atmospheric circulation. Surface variables like temperature and precipitation are largely correlated with weather regimes .

The North Atlantic weather regimes were computed with the procedure of , with the NCEP reanalysis. The first 10 principal components of SLP (large region in Fig. b) were classified with a k-means algorithm onto four classes over a reference period between 1970 and 2010. The procedure was repeated 100 times with random k-means initialization. Then we classified the resulting 100×4 k-means weather regimes in order to determine the most probable classification. This heuristic procedure increases the robustness of the obtained weather regimes. Figure  shows four weather regimes for each season (winter and summer) that are coherent with the literature .

The winter weather regimes are the negative phase of the North Atlantic oscillation (NAO-), Atlantic Ridge (AR), Scandinavian blocking (BLO), and Zonal (ZO). The summer weather regimes are the negative phase of the NAO (NAO-), Zonal (ZO), Scandinavian blocking (BLO), and Atlantic low (AL). The regimes are not the same in both seasons due to the seasonality of the large-scale atmospheric circulation.

For each day (in winter and summer) between 1948 and 2019, we classified the SLP by minimizing the root mean square to four reference (1970–2010) weather regimes.

For each day t (within a given season), we considered the analog dates of all N=100 simulations between t and t+T and the corresponding classification into weather regimes. Then we determined the most frequent weather regime of the N member ensemble forecast between t and t+T. We hence obtained time series on the most likely weather pattern that dominates in the ensemble forecast between t and t+T.

We evaluated the influence of the dominating weather regimes on the SWG forecast quality by plotting the probability distribution of CRPS values conditioned on the weather regimes. This is done separately for “good” forecasts (low CRPS values) and “poor” forecasts (high CRPS values).

We identified two classes of predictability from CRPS values:

Low predictability is related to high values of CRPS that exceed the 75th quantile.

As an example, we illustrate the behavior of the trajectories in Orly for the summer and winter of 2002. Figure  shows the observed and simulated values of precipitation for lead times of 5 and 10 d for summer (June–July–August: JJA) and winter (December–January–February: DJF), for Orly precipitation data. We observe significantly positive correlations between observed values and the median of the forecasts for the four data sets as represented in Table . The correlation is generally smaller in the summer than in the winter. The correlation skill is low for some extreme values of precipitation. For a lead time of 10 d, SWG simulation still shows a capacity to predict precipitation, in particular for winter with a correlation equal to 0.23 (Orly), 0.30 (Berlin), 0.43 (Madrid), and 0.31 (Toulouse).

We observe that the 5th and 95th quantiles of the simulations include the different values of observations. This heuristically confirms the good skill of SWG to forecast precipitation from Z500 for various seasons (winter and summer) in several locations for T=5 and T=10 d lead times.

The difference in the forecast correlation skills between the four studied locations may be related to the variation of the local climate from one region to another. The studied areas are in different climate types according to the Köppen–Geiger climate classification . From the southwestern side of Europe, Madrid is in the arid zone of the classification , which indicates that convective rains are less frequent, and the origin of precipitation might be the result of humidity coming from the Atlantic. Conversely, Berlin is located in a cold zone characterized by warm summer and the absence of a dry season ; the precipitation could be the result of both, convective rains and Atlantic humidity.

In this paper, we decided (for simplicity) to use the same analogs to forecast precipitation for those four stations as discussed in Sect. . A refinement of the analog regions would be necessary when focusing on Madrid vs. Berlin.

The CRPSS and correlation skill scores are computed for the four studied stations (Berlin, Madrid, Orly, and Toulouse), as shown in Fig.  and for lead times from 5 to 20 d.

In this paper, we chose to present the results for summer and winter to highlight the capacity of the SWG to forecast the precipitation in extreme seasons. We focus on January and July in order to show the skill of the SWG in predicting precipitation in different conditions.

The CRPSS against the persistence and climatology references show positive values for lead times of up to 20 d (Fig. ). The values of CRPSS against the persistence reference (represented by squares) decrease with lead times in winter for the different studied areas, showing high values over 5 d. However, for summer, we notice that the values of CRPSS against persistence increase with lead time, with high values over 20 d except for Berlin. This indicates that the SWG forecast is still better than the persistence forecast (the average of the CRPS of SWG is smaller than the average of the CRPS of the persistence) for lead times of 20 d in the summer. This could be explained by the fact that summer precipitation in Orly (51 % of the time, on average) comes in clusters contrary to precipitation in Berlin. Indeed, we computed the seasonal frequency of precipitation (defined as the number of days when precipitation exceeds 0.5 mm d-1). We found that for Berlin, precipitation exceeding 0.5 mm d-1 is more frequent than in the other stations (close to 50 % of the time for both seasons).

This means that a persistence forecast for Orly is likely to be skillful, even for longer lead times, especially in the summer. Therefore, the trends in CRPSS values for different lead times are probably due to the intrinsic time persistence of local precipitation.

The CRPSS against the climatology reference (triangles in Fig. ) shows lower values compared with the CRPSS against persistence reference, although they are positive for all lead times and for both seasons. However, we notice that for a short lead time the SWG is better than the climatology.

The correlation skill is positive for both seasons but higher in winter (January) than in summer (July). For a lead time of 5 d, the correlation is equal to 0.59 for Madrid, 0.50 for Berlin, and 0.40 for Toulouse. For a lead time of 10 d, it is equal to 0.42 for Madrid, 0.30 for Berlin, and 0.41 for Toulouse.

The SWG was tested by to forecast temperature in western Europe. Comparing the performance of the SWG to forecast those different meteorologic variables, we noticed that the model shows good performance to forecast the temperature in the winter; also the best performance of the model is at a lead time of 5 d. We find that the skill scores (CRPSS and correlation) decrease with lead times. The forecast skill of the SWG shows variability from one location to another. However, the model was able to forecast temperature until 40 d in Berlin, Orly, and Toulouse with positive skill scores.

From a visual inspection of the CRPSS and correlations, we chose to focus on lead times of T=5 d, for which the correlation exceeds 0.5 in the winter. It is rather low in the summer, due to convective events leading to a high precipitation variability (from no rain to very high values). Correlation scores become barely significant for lead times of 20 d, so that, like temperature, the SWG should not be used beyond that horizon.

We investigated the role of North Atlantic weather patterns defined in Sect.  (Fig. ) on the forecast skill of the SWG precipitation simulations.

We started by comparing the frequencies of the weather regimes from the observations and the most frequent weather regime found in SWG simulations for a given lead time T=5 d. We found that the percentages are very similar (Fig. ). This means that the weather regimes of the simulated trajectories do not yield major biases for the summer or winter seasons.

Then we looked at the relation between weather regimes and CRPS values by using the most frequent weather regime within T days and the two classes of quantiles of the CRPS that related to good quality of forecast (attributed to low values of CRPS ≤q25) and poor quality of forecast (attributed to high values of CRPS ≥q75). This relation is represented in Fig.  for Orly and for the rest of the studied stations in Fig. . We found a small influence of specific weather regimes on the CRPS distribution for summer.

The weather regime signal for “good” forecasts depends on the season and the considered station. When the forecast has a low CRPS value (for Orly), we find that the Scandinavian blocking regime slightly dominates (green bar in Fig. a, b). This is also the case for Berlin (in winter) and Toulouse (Fig. b, j). The low CRPS values in Madrid are obtained for the Atlantic Ridge regime (Fig. f).

The weather regime signal for “poor” forecasts also yields a dependence on the season and station. Higher CRPS values are obtained with the Zonal regime in the summer for Orly (red line in Fig. c) and Toulouse. The Atlantic Ridge regime favors high CRPS values (i.e., poor forecasts) for Madrid in winter Fig. h. The Scandinavian blocking favors high CRPS values for Berlin in winter and summer (green line in Fig. c and d). The different impacts of the weather regimes on the studied areas are related to the position of the high- and low-pressure regions of each weather regime in the studied areas.

This relation between predictability (or the CRPS distribution) and weather regimes, albeit weak, is consistent with previous work of . Similar relations were found between weather regimes over Europe and the temperature in a recent study by . We found that the sensitivity of the forecast to weather regime is larger for low values of CRPS and in winter. The sensitivity of forecast skill to weather regimes is rather small on average, even for small lead times (T=5 d).

We first compared the CRPSS of SWG forecasts for winter and summer with the CRPSS of ECMWF forecasts.

The CRPSS of the ECMWF forecast is computed for different lead times going from 1 to 10 d for precipitation over the region 12.5∘ W–42.5∘ E; 35.0–75.0∘ N . It uses the climatology as a reference . The values of CRPSS for Europe for 2020 decrease in accordance with lead times . The CRPSS of ECMWF is about 0.16 in summer (JJA) and 0.25 in winter (DJF) for a lead time of T=5 d . The CRPSS of SWG simulations for a lead time of T=5 d is shown in Table . The values suggest that the predictive skill of SWG is qualitatively promising for short lead times, compared with ECMWF forecasts. However, we have to mention that the values of CRPSS for ECMWF are computed over all of Europe for both seasons , while with the SWG we are doing a forecast for local stations.

We made a quantitative comparison between the two forecasts for the different lead times. We computed the CRPS for the ECMWF forecast. Then we used the Kolmogorov–Smirnov (KS) test (, chap. 1) to compare the probability distributions of the CRPS of SWG and ECMWF forecasts. The null hypothesis supposes that the CRPS of ECMWF and SWG forecasts have the same distribution. The null hypothesis of the KS test was rejected; this means that the two time series do not have the same distribution, with a p value =0.11. A similar result was found by , where they compared the efficiency between ECMWF and CNRM forecasts.

We found that 80 %, 39 %, 50 %, and 40 % of the CRPS of SWG forecast are equal to zero for, respectively, Orly, Berlin, Madrid, and Toulouse, for a lead time of T=5 d Fig. , which shows the capacity of the SWG to simulate rain events well. One notable difference between SWG and ECMWF forecasts is that although the proportion of CRPS values close to zero is higher for ECMWF, the CRPS for the worst forecasts is much higher than those of SWG. Indeed, we noticed that the time average of CRPS of ECMWF (vertical blue lines) and SWG (red vertical lines) for T=5 d are close, with higher values for ECMWF (Fig. ). However, the median CRPS of ECMWF is smaller compared with the SWG (dashed vertical lines in Fig. ). Finally, we computed the CRPSS for ECMWF forecasts taking as a reference the CRPS of SWG (Table ). We hence computed the CRPSS of ECMWF forecast by normalizing the CRPS by the CRPS of the SWG forecast in Eq. ().

This new ECMWF CRPSS evaluates the added value of the ECMWF forecast over the SWG forecast. We found that the ECMWF forecast has no improvement over the SWG forecast for the different lead times because the CRPSS values are negative. At T=5 d, we noticed that the improvement is negligible for Orly and Berlin, while it is much better for Madrid. However, for Toulouse, the ECMWF forecast still has better skills for lead times of T=5 and 10 d. For a lead time of T=10 d, the improvement of the SWG forecast over the ECMWF is significant, particularly for Berlin and Madrid. There is a major improvement for a lead time of T=20 d for Orly and Berlin.

This confirms the relatively good skill of the SWG to forecast precipitation, compared with ECMWF. This could be explained by the difference in the average of the CRPS of the two forecasts. Indeed, as we mentioned before, the ECMWF forecast yields the best skill scores for small values of precipitations (<2 mm d-1). We further illustrate those comparisons in Fig.  and Table .