GAN–argcPredNet v1.0: a generative adversarial model for radar echo extrapolation based on convolutional recurrent units

Precipitation nowcasting plays a vital role in preventing meteorological disasters, and Doppler radar data act as an important input for nowcasting models. When using the traditional extrapolation method it is difficult to model highly nonlinear echo movements. The key challenge of the nowcasting mission lies in achieving high-precision radar echo extrapolation. In recent years, machine learning has made great progress in the extrapolation of weather radar echoes. However, most of models neglect the multi-modal characteristics of radar echo data, resulting in blurred and unrealistic prediction images. This paper aims to solve this problem by utilizing the features of a generative adversarial network (GAN), which can enhance multi-modal distribution modeling, and design the radar echo extrapolation model GAN–argcPredNet v1.0. The model is composed of an argcPredNet generator and a convolutional neural network discriminator. In the generator, a gate controlling the memory and output is designed in the rgcLSTM component, thereby reducing the loss of spatiotemporal information. In the discriminator, the model uses a dual-channel input method, which enables it to strictly score according to the true echo distribution, and it thus has a more powerful discrimination ability. Through experiments on a radar dataset from Shenzhen, China, the results show that the radar echo hit rate (probability of detection; POD) and critical success index (CSI) have an average increase of 21.4 % and 19 %, respectively, compared with rgcPredNet under different intensity rainfall thresholds, and the false alarm rate (FAR) has decreased by an average of 17.9 %. We also found a problem during the comparison of the result graph and the evaluation index. The recursive prediction method will produce the phenomenon that the prediction result will gradually deviate from the true value over time. In addition, the accuracy of high-intensity echo extrapolation is relatively low. This is a question worthy of further investigation. In the future, we will continue to conduct research from these two directions.

1 Introduction

Precipitation nowcasting refers to the prediction and analysis of rainfall in the target area over a short period of time (0–6 h) (Bihlo, 2019; Luo et al., 2020). The important data needed for this work come from Doppler weather radar with high temporal and spatial resolution (Wang et al., 2007). Relevant departments can issue early warning information through accurate nowcasting to avoid loss of life and destruction of infrastructure (Luo et al., 2021). However, this task is extremely challenging due to its very low tolerance for time and position errors (Sun et al., 2014).

The existing nowcasting systems mainly include two types, numerical weather prediction (NWP) and radar echo extrapolation (Chen et al., 2020). The widely used optical flow method several problems, such as its poor capture quality in fast echo change regions, the high complexity of the algorithm and its low efficiency (Shangzan et al., 2017). Since echo extrapolation can be considered a time series image-prediction problem, these shortcomings of the optical flow method are expected to be solved by using a recurrent neural network (RNN) (Giles et al., 1994).

With the continuous development of deep learning, more and more neural networks have been applied to the field of nowcasting. Forecast models such as ConvLSTM and EBGAN-Forecaster show that this new method's extrapolation effect is better than that of optical flow method (Shi et al., 2015; Chen et al., 2019). However, these models still have the problem of blurred and unrealistic prediction images (Tian et al., 2020; Xie et al., 2020; Jing et al., 2019). One of the main reasons is that radar echo maps are typically multi-modal data acquired by multiple sensors and different stations; some algorithms ignore this feature of radar echo maps, using the mean square error and mean absolute error as the loss function, which is better suited to a unimodal distribution.

The paper proposes a generative adversarial network-argcPredNet (GAN–argcPredNet) network model, which aims to solve this problem through GAN's ability to strengthen the characteristics of multi-modal data modeling. The generator adopts the same deep coding–decoding method as PredNet to establish a prediction model, and uses a new structure of convolutional long short-term memory (LSTM) as a predictive neuron, which can effectively reduce the loss of spatiotemporal information compared with rgcLSTM. The deep convolutional network is used as the discriminator to classify the distribution, and the dual-channel input mechanism is used to strictly judge the distribution of real radar echo images. Finally, based on the weather radar echo dataset, the generator and the discriminator are alternately trained to make the extrapolated radar echo map more real and precise.

2 Related work

2.1 Sequence prediction networks

The essence of radar echo image extrapolation is the problem of sequence image prediction, which can be solved by implementing an end-to-end sequence learning method (Shi et al., 2015; Sutskever et al., 2014). ConvLSTM introduces a convolution operation into the conversion of the internal data state of the LSTM, effectively utilizing the spatial information of the radar echo data (Shi et al., 2015). However, the trajectory gated recurrent unit (TraijGRU) has also been proposed as a solution (Shi et al. 2017) since the location-invariant nature of the convolutional recursive structure is inconsistent with the natural change motion. A GRU (gated recurrent unit), as a kind of recurrent neural network, performs a similar function to LSTM but is computationally cheaper (Group, 2017). Similarly, ConvGRU introduces convolution operations inside the GRU to enhance the sparse connectivity of the model unit and is used to learn video spatiotemporal features (Ballas et al., 2015). The RainNet network learns the movement and evolution of radar echo based on the U-NET convolutional network for extrapolation prediction (Ayzel et al., 2020). PredNet is based on a deep coding framework and adds error units to each network layer that can transmit error signals like the human brain structure (Lotter et al., 2016). In order to increase the depth of the network and the connections between modules, Skip-PredNet further introduces skip connections and uses ConvGRU as the core prediction unit. Experiments show that its effect is better than the TrajGRU benchmark (Sato et al., 2018). Although these networks can achieve echo prediction, they have problems with both blurring and producing unrealistic extrapolated images.

2.2 GAN-based radar echo extrapolation

The generative adversarial network (GAN) consists of two parts: a generator and a discriminator (Goodfellow et al., 2014). GAN can be an effective model for generating images. Using an additional GAN loss, a model can better achieve multi-modal data modeling and each of its outputs will be clearer and more realistic (Lotter et al., 2016). Multiple complementary learning strategies show that generative adversarial training can maintain the sharpness of future frames and solve the problem of lack of clarity in prediction (Michael et al., 2015). In this regard, the extrapolators built a generative adversarial network to solve the problem of extrapolated image blur by trying to use this adversarial training to extrapolate more detailed radar echo maps (Singh et al., 2017). Similarly, an adversarial network with ConvGRU as the core was proposed, mainly to solve the problem of ConvGRU's inability to achieve multi-modal data modeling (Tian et al., 2020). There are also researchers working on the idea of a four-level pyramid convolution structure that propose four pairs of models to generate an adversarial network for radar echo prediction (Chen et al., 2019). It should be noted that the traditional GAN network has the problem that it uses unstable training, which will cause the model unable to learn. Therefore, the design of the nowcasting model should be based on a stable and optimized GAN network.

3 Model

In this section, we describe the model both overall and in detail. Section 3.1 introduces the overall structure and training process of the model. In Sect. 3.2, we describe the structure of the argcPredNet generator and focus on the argcLSTM neuron. In Sect. 3.3, we introduce the design of the discriminator and the loss function of the model.

3.1 GAN–argcPredNet model overview

Radar echo extrapolation refers to the prediction of the dissipation and distribution of future echoes based on the existing radar echo sequence diagram. If the problem is formulated, then each echo map can be regarded as a tensor, i.e., $x\in R^{\mathrm{W}\times \mathrm{H}\times \mathrm{C}}$, where W, H and C represent the width, height and number of channels, respectively, and R represents the feature area. If M represents a given sequence of echo maps and N is used to predict the most likely changes in the future, this problem can be expressed as Eq. (1). This paper sets the input sequence M and output sequence N to 5 and 7, respectively.

$$\begin{array}{}\text{(1)}& \begin{array}{rl}{\widehat{x}}^{t+\mathrm{1}},\mathrm{\dots },{\widehat{x}}^{t+N}& =\begin{array}{c}\mathrm{argmax}\\ x^{t+\mathrm{1}},\mathrm{\dots },x^{t+N}\end{array}\\ & p(x^{t+\mathrm{1}},\mathrm{\dots },x^{t+N}|x^{t-M+\mathrm{1}},\mathrm{\dots },x^t)\end{array}\end{array}$$

Unlike other forecasting models, GAN–argcPredNet uses WGAN-gp (Wasserstein generative adversarial network with gradient penalty) as a predictive framework. The model solves the problem of training instability through gradient penalty measures (Gulrajani et al., 2017). Our model mainly includes two parts: generator and discriminator. As shown in Fig. 1, the generator is composed of argcPredNet, which is responsible for learning the potential features of the data and simulating the data distribution to generate prediction samples. Following this, the predicted samples and the real samples are input into the discriminator to make a judgment, where the real data are judged to be true, and thus the predicted data are judged to be false. Finally, we use the Adam optimizer for training the adversarial loss and then update the parameters of the discriminator, optimize the loss function of the generator once every five updates and complete the update of the generator parameters. The algorithm flow is shown in Table 1 (Gulrajani et al., 2017).

Figure 1 The schematic of the GAN–argcPredNet architecture.

Table 1 GAN–argcPredNet training algorithm flow.

3.2 argcPredNet generator

3.2.1 argcLSTM

The internal structure of the argcLSTM neuron used in the model is shown in Fig. 2. In order to provide better feature-extraction capabilities, the structure contains two trainable gating units: the forget gate f^(t) and the input gate g^(t). The latter can calculate the weight of the current state independently and complete the feature retention of the input information. The peephole connection from the unit state to the forget gate is removed. This operation does not have a big impact on the result, but it simplifies the redundant parameters. The complete definition of the argcLSTM unit is as follows (Eqs. 2–5).

$$\begin{array}{}\text{(2)}& {\displaystyle }{f}^{\left(t\right)}& {\displaystyle }=\mathit{\sigma }\left(W_{fx}\ast x^{\left(t\right)}+U_{fh}\ast h^{\left(t-\mathrm{1}\right)}+b_f\right)\text{(3)}& {\displaystyle }{g}^{\left(t\right)}& {\displaystyle }=\mathit{\sigma }\left(W_{gx}\ast x^{\left(t\right)}+U_{gh}\ast h^{\left(t-\mathrm{1}\right)}+b_g\right)\text{(4)}& {\displaystyle }{\stackrel{\mathrm{̃}}{C}}^{\left(t\right)}& {\displaystyle }=\tanh \left(W_{ch}\ast h^{\left(t-\mathrm{1}\right)}+W_{cx}\ast x^{\left(t\right)}+b_c\right)\text{(5)}& {\displaystyle }{C}^{\left(t\right)}& {\displaystyle }=f^{\left(t\right)}\circ C^{\left(t-\mathrm{1}\right)}+g^{\left(t\right)}\circ {\stackrel{\mathrm{̃}}{C}}^{\left(t\right)}\text{(6)}& {\displaystyle }{h}^{\left(t\right)}& {\displaystyle }=g^{\left(t\right)}\circ \tanh \left(C^{\left(t\right)}\right)\end{array}$$

In Eqs. (2)–(6), ∗ represents convolution operation; ∘ represents the Hadamard product; σ represents sigmoid nonlinear activation function' f^(t) and g^(t) represent the forget gate and update gate, respectively; and x^(t), h^(t), and C^(t) represent the input, hidden state, and unit state at time t, respectively.

Figure 2 The internal structure of argcLSTM.

3.2.2 argcPredNet

The argcPredNet generator has the same structure as PredNet, which is composed of a series of repeatedly stacked modules, with a total of three layers. The difference is that argcPredNet uses argcLSTM as the prediction unit. As shown in Fig. 3, each layer of the module contains the following four units: the input convolutional layer (A_l), the recurrent representation layer (R_l), the prediction convolutional layer (${\widehat{A}}_l$) and the error representation layer (E_l).

Figure 3 Module expansion diagram of layer l at time t.

The recursive prediction layer uses the argcLSTM loop unit, which is used to generate the prediction of the next frame and the input of ${\widehat{A}}_l$ and A_l+1. The network uses error calculation, where E_l will output an error representation, and the error representation is passed forward through the convolutional layer to become the input of the next layer A_l+1. The hidden state of the recurrent unit $R_l^t$ is updated according to the output of $E_l^{t-\mathrm{1}}$, $R_l^{t-\mathrm{1}}$ and the up-sampled $R_{l+\mathrm{1}}^t$. For A_l, the input of the lowest target, namely A₀, is set to the actual sequence itself. When l>0, the input of A_l is the lower error signal E_l−1, which results from convolution calculation, linear rectification function (RELU) activation and the maximum pooling layer. The complete update rules are shown in Eqs. (7) to (10). The specific parameter settings of the generator are shown in Table 2. The numbers 1, 128, 128 and 256 represent the number of filters used from the first layer to the fourth layer.

$$\begin{array}{ll}{\displaystyle }{A}_l^t& {\displaystyle }=\left\{\begin{array}{l}{x}_t\\ \mathrm{MAXPOOL}\left(\mathrm{RELU}\left(\mathrm{CONV}\left(E_{\mathrm{l}-\mathrm{1}}^t\right)\right)\right)\end{array}\right.\\ \text{(7)}& {\displaystyle }& {\displaystyle }\begin{array}{c}\text{if}l=\mathrm{0}\\ \mathrm{0}<l<L\end{array}\text{(8)}& {\displaystyle }{\widehat{A}}_l^t& {\displaystyle }=\mathrm{RELU}\left(\mathrm{CONV}\left(R_l^t\right)\right)\text{(9)}& {\displaystyle }{E}_l^t& {\displaystyle }=\left[\mathrm{RELU}\left(A_l^t-{\widehat{A}}_l^t\right);\mathrm{RELU}\left({\widehat{A}}_l^t-A_l^t\right)\right]{\displaystyle }{R}_l^t& {\displaystyle }=\left\{\begin{array}{l}\mathrm{argcLSTM}\left(E_l^{t-\mathrm{1}},R_l^{t-\mathrm{1}}\right)\\ \mathrm{argcLSTM}\left(E_l^{t-\mathrm{1}},R_l^{t-\mathrm{1}},\mathrm{UPSAMPLE}\left(R_{l+\mathrm{1}}^t\right)\right)\end{array}\right.\\ \text{(10)}& {\displaystyle }& {\displaystyle }\begin{array}{c}\text{if}l=L\\ \mathrm{0}<l<L\end{array}\end{array}$$

Table 2 Generator parameter settings.

3.3 Discriminator and loss

3.3.1 Convolutional discriminator

The purpose of the discriminator is to recognize images, which similar to the purpose to the classifier. In the GAN–argcPredNet model, a double-channel convolutional neural network (DC-CNN) network is designed for discrimination. The process is shown in Fig. 4. It is a four-layer convolution model with a dual-channel input method.

The DC-CNN network extracts a pair of images from the three pairs of images, inputs them to the fully connected layer through a four-layer convolution transformation and finally generates a probability output through the Sigmoid function, indicating the possibility that the input image is from a real image. When the input is a real image, the discriminator will maximize the probability, and thus the value will approach 1. If the input is a generator-synthesized image, the discriminator will minimize the probability, and thus the value will approach −1. The specific parameter settings of the discriminator are shown in Table 3.

Figure 4 The DC-CNN structure.

Table 3 Discriminator parameter settings.

3.3.2 Loss function

The generative adversarial network relies on the distribution of simulated data to generate images. It can retain more echo details, thereby realizing the modeling of multi-modal data. A gradient penalty term is added to GAN–argcPredNet, and the loss function of the discriminator is shown in Eq. (11).

$$\begin{array}{}\text{(11)}& L_D=D\left(\stackrel{\mathrm{̃}}{x}\right)-D\left(x\right)+\mathit{\lambda }{\left({∥{\mathrm{\nabla }}_{\widehat{x}}D\left(\widehat{x}\right)∥}_{\mathrm{2}}-\mathrm{1}\right)}^{\mathrm{2}}\end{array}$$

The generator has the following loss function (Eq. 12):

$$\begin{array}{}\text{(12)}& L_G=E_{\stackrel{\mathrm{̃}}{x}\sim P_{\mathrm{g}}}\left[D\left(\stackrel{\mathrm{̃}}{x}\right)\right]-E_{x\sim P_{\mathrm{r}}}\left[D\left(x\right)\right].\end{array}$$

The model has the following maximum–minimum loss function (Eq. 13):

$$\begin{array}{ll}{\displaystyle }\begin{array}{l}min\\ G\end{array}\begin{array}{l}max\\ D\end{array}V\left(D,G\right)& {\displaystyle }=E_{\stackrel{\mathrm{̃}}{x}\sim P_{\mathrm{g}}}\left[D\left(\stackrel{\mathrm{̃}}{x}\right)\right]-E_{x\sim P_{\mathrm{r}}}\left[D\left(x\right)\right]\\ \text{(13)}& {\displaystyle }& {\displaystyle }+\mathit{\lambda }{E}_{\widehat{x}\sim P_{\widehat{x}}}{\left({∥{\mathrm{\nabla }}_{\widehat{x}}D\left(\widehat{x}\right)∥}_{\mathrm{2}}-\mathrm{1}\right)}^{\mathrm{2}},\end{array}$$

where $\stackrel{\mathrm{̃}}{x}$ represents the distribution of generated samples, P_g represents the set of generated sample distributions, x represents the distribution of real samples and P_r represents the set of real sample distributions. The third term is the penalty item of the gradient penalty mechanism. In the penalty term, $\widehat{x}$ represents the actual data and generation of a new sample formed by random sampling between data. $P_{\widehat{x}}$ represents a set of randomly sampled samples. λ is a hyperparameter that represents the coefficient of the penalty term, and the value in the model is set to 10.

4 Experiments

In order to verify the effectiveness of the model, the paper uses the radar echo data from January to July 2020 in Shenzhen, China, to conduct experiments on the four models of ConvGRU, rgcPredNet, argcPredNet, and GAN–argcPredNet. All experiments are implemented in Python and based on the Keras deep-learning library with TensorFlow as the back end for model training and testing.

4.1 Dataset description

This experiment uses the radar echo data of Shenzhen, China. The dataset is made up of rain images after quality control. The reflectivity range is 0–80 dBZ, the amplitude limit is between 0 and 255, and the data are collected every 6 min, with a total of one layer. The height above sea level is 3 km. A total of 600 000 echo images were collected, of which 550 000 were used as the training set and 50 000 were used as the test set. Each set of data contained 12 consecutive images. The horizontal resolution of the radar echo maps is 0.01^∘ (about 1 km), the number of grids is 501×501 (i.e., an area of about 500 km × 500 km) and the image resolution is 96×96 pixels.

4.2 Evaluation metrics

In order to evaluate the accuracy of the model for precipitation nowcasting, the experiment uses three evaluation indicators to evaluate the prediction precision of the model, critical success index (CSI, Eq. 14), false alarm rate (FAR, Eq. 15) and hit rate (probability of detection, POD; Eq. 16).

$$\begin{array}{}\text{(14)}& {\displaystyle }\mathrm{CSI}& {\displaystyle }={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}+\mathrm{FP}}}\text{(15)}& {\displaystyle }\mathrm{FAR}& {\displaystyle }={\displaystyle \frac{\mathrm{FN}}{\mathrm{TP}+\mathrm{FN}}}\text{(16)}& {\displaystyle }\mathrm{POD}& {\displaystyle }={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}}\end{array}$$

In the formula, TP indicates that both the predicted value and the true value reach the specified threshold, FN means that the true value reaches the specified threshold but that the predicted value has been not reached, and FP indicates that the true value has not reached the specified threshold but that the predicted value has reached the specified threshold.

Figure 5 CSI index score.

Figure 6 POD index score.

Figure 7 FAR index score.

4.3 Results

The experiment comprehensively evaluates the prediction accuracy of precipitation with different thresholds. The radar reflectivity and rainfall intensity refer to the Z–R relationship (Shi et al., 2017). The calculation formula is as follows:

$$\begin{array}{}\text{(17)}& Z=\mathrm{10}\log a+\mathrm{10}b\log R.\end{array}$$

In this paper, a is set to 58.53, b is set to 1.56, Z represents the intensity of radar reflectivity, R represents the intensity of rainfall, and the corresponding relationship between rainfall and rainfall level is given in Table 4 (Shi et al., 2017).

Table 4 Rain level

Figures 5, 6 and 7 compare the CSI, POD and FAR index scores, respectively in detail for each model at different rainfall thresholds.

Figure 8 Four prediction examples for the precipitation nowcasting problem. From top to bottom: ground truth frames, prediction by GAN–argcPredNet, prediction by argcPredNet, prediction by rgcPredNet, prediction by ConvGRU.

This result is calculated based on 50 000 test pictures (more than 4000 test sets), which is taken as a representative number. It can be seen that when the rainfall rate increases from 0.5 to 30 mm h⁻¹, GAN–argcPredNet always performs best (by a significant margin advantage), argcPredNet is second, and the ConvGRU model is the worst. Another point worth noting is that as the rainfall intensity increases, the performance of all models shows a downward trend. In the comparison of CSI indicators, GAN–argcPredNet leads the rest of the models when the rainfall rate is lower than when the rainfall rate is lower than 30 mm h⁻¹. When the rainfall level is 30≤  R, its leading advantage is the weakest. ArgcPredNet only leads rgcPredNet by a slight margin, and its performance with rgcPredNet is not as good as ConvGRU in the range of 2–5 mm h⁻¹. For POD indicators, GAN–argcPredNet performs best, and its leading advantage is more prominent. The performance of argcPredNet is not so outstanding as it is almost the same as argcPredNet, but the index of the two is always better than ConvGRU. For the FAR score, the performance of GAN–argcPredNet is still the best, while argcPredNet and rgcPredNet are worse than ConvGRU in the range of 2–5 mm h⁻¹, and the score for the rain rate 0.5–10 mm h⁻¹ interval is slightly lower than that of GAN–argcPredNet.

To compare the three methods more intuitively, Fig. 8 shows the image prediction results of the three models on the same piece of test data.

Compared with the other three models, GAN–argcPredNet generates better image clarity and shows more detailed features on a small scale. The contrast between the areas marked by the red circle in Fig. 8 is more obvious. GAN–argcPredNet made the best prediction of the shape and intensity of the echo. The area selected by the rectangle mainly shows the echo changes in the northern region within 30 min. Both models correctly predict the movement of the echo to a certain extent, and the prediction process shown by GAN–argcPredNet is the most complete. In some mixed intensity and edge areas, our model clearly predicts the echo intensity information, which clearly shows the effect of the confrontation training.

In order to compare the prediction results more specifically, the experiment uses mean square error (MSE) and mean structural similarity (MSSIM) to evaluate the quality of the generated images (Wang et al., 2004). The MSE and MSSIM index scores of the images generated by each model are shown in Table 5. ConvGRU has the lowest two indexes. Although the MSE index of rgcPredNet is slightly lower than that of the argcPredNet and GAN–argcPredNet models, the MSSIM index of the argcPredNet and GAN–argcPredNet models is 0.066 and 0.109 higher than that of the rgcPredNet network model, respectively.

Table 5 MSE and MSSIM index scores of each model.

5 Conclusions

This study demonstrated a radar echo extrapolation model. The main innovations are summarized as follows. First, the argcPredNet generator is established based on the time and space characteristics of radar data. The argcPredNet generator can predict future echo changes based on historical echo observations. Second, our model uses adversarial training methods to try to solve the problem of blurry predictions.

Based on the evaluation indicators and qualitative analysis results, GAN–argcPredNet has achieved excellent results. Our model can reduce the prediction loss in a small-scale space so that the prediction results have more detailed features. However, the recursive extrapolation method causes the error to accumulate as time goes by, and the prediction result deviates more and more from the true value. In addition, when the amount of high-intensity echo data is small, the prediction of high-risk and strong convective weather through machine learning is also a problem that we are very concerned about because it is more realistic. Therefore, we will carry out research into these two issues in the future.