MSDM:a machine learning model for precipitation nowcasting over east China using multi-source data

East China is one of the most economically developed and most densely populated areas in the world. Due to its special geographical location and climate, East China is affected by different weather systems like monsoon, shear line, typhoon and extratropical cyclone, in the imminent future the rainfall rate affected by which is difficult to precisely predict. Traditional physics-based methods like Numerical Weather Prediction (NWP) tend to perform poorly for the nowcasting 10 problem due to its spinup issue. Meanwhile, various meteorological stations are distributed here, generating a large amount of observation data every day, which has a great potential to be applied to data-driven methods. Thus, it is important to train a data-driven model from scratch that suitable to the specific weather situation of East China. We collect three kinds of data (radar, satellite, precipitation) in flood season from 2017 to 2018 of this area and preprocess them into ndarray (256×256) that cover East China with a domain of 12.8×12.8°.The Multi-Source Data Model (MSDM) which we developed combines 15 the Optical flow, Random forest and Convolutional Neural Network (CNN). It treats the precipitation nowcasting task as an image-to-image problem, which takes radar and satellite data with a interval of 30 minutes as inputs and predicts radar echo intensity at a lead time of 30 minutes. To reduce the smoothing caused by convolution, we use Optical flow to predict satellite data in the following 120 minutes. The predicted radar echo from MSDM together with satellite data from Optical flow are recursively implemented in MSDM to achieve 120 minutes lead time. The predictions from MSDM are comparable to those 20 of other baseline models with a high temporal resolution of 6 minutes. To solve the blurry image problems, we applied a modified SSIM as a loss function. Furthermore, we use Random forest with predicted radar and satellite data to estimate the rainfall rate, the results outperform those of the traditional Z-R relationship. The experiments confirm that machine learning with multi-source data provides more reasonable predictions and reveals a better non-linear relationship between radar echo and precipitation rate. Besides the algorithms will be developed, exploiting the potential of multi-source data will bring more 25 improvements.


Introduction
In recent years, deep learning and machine learning have achieved great advances with big data. The tremendous meteorological data are produced every day, which perfectly matches these novel data-driven AI approaches. Quantitative https://doi.org/10.5194/gmd-2020-363 Preprint. Discussion started: 21 December 2020 c Author(s) 2020. CC BY 4.0 License.
Precipitation Nowcasting(QPN) by using Radar Echo Extrapolation(REE) have become popular recently. (Tran and Song, 30 2019a). Precipitation Nowcasting makes the prediction of rainfall intensity in the following several hours. Based on various data with high spatio-temporal resolutions, the AI precipitation prediction can be relatively accurate compared with traditional NWP methods. Agrawal et al.(Agrawal et al., 2019) treated the precipitation nowcasting as an image-to-image problem. They employed the U-Net to predict the change of radar echo for QPN, which is superior to NOAA's HRRR numerical prediction when the prediction time is within 6 hours. Sonderby et al. (Sønderby et al., n.d.) proposed a MetNet to discover the weather 35 pattern from radar and satellite data which can predict the next 8 hours precipitation with a resolution of 1 kilometer in 2minute intervals. Shi et al.(Shi et al., n.d.) used the Convolutional Long Short-Term Memory (ConvLSTM) to predict the spatiotemporal sequences of the precipitation. And they also provide the first benchmark as well as a new TrajGRU model to capture the spatiotemporal correlations (Shi et al., n.d.). Also, in the field of video prediction, Wang et al. proposed various recurrent networks like PredRNN++ (Wang et al., 2018), MIM (Wang et al., 2019b), E3D-LSTM (Wang et al., 2019a). However, 40 their work is based on a slight modification of existing techniques demanding massive computing resource to train and haven't been applied to the numerous meteorological data.
Computer vision techniques have long been used in object detection, video prediction, and human motion prediction, etc. Tran and Song (Tran and Song, 2019b) used image quality assessment techniques as a new loss function instead of the common mean squared error(MSE), which will mislead the process of training and generate the blurry image. Optical flow methods 45 simply describe the position and velocity of the radar echo with a constant velocity. Ayzel et al (Ayzel et al., 2019) designed an advanced model based on the multiple optical flow algorithm for QPN, but it still performs badly in the prediction of onset and decay of precipitation systems.
Hence, to make full use of massive meteorological data combining Optical Flow methods and Deep learning are conducted to predict the QPN based on the characteristics of multi-source data, such as radar echo, Infrared satellite data, and observation 50 data et al., Due to the limit of computing resources, the aim of this paper is not to achieve higher resolution or accuracy of the prediction but to propose a method of combining Optical Flow and CNN with radar echo data, satellite data, and automatic ground observation data to make QPN.
The dataset and methods used for this study are described in section 2. Section 3 shows the results. Section 4 draws conclusions and discusses some possible future work. 55

Dataset
The spatial and temporal distribution characteristics of precipitation are related to many factors like terrain, atmospheric circulation, and climatic conditions, etc. To train the Deep learning model to learn the precipitation characteristics of East China, we collected multi-source observation data of the flood season (May to September) for a total of 306 days from 2017 60 to 2018. Due to the missing radar data from May 1 to 9 and September 26 to 30, 2018, the radar data is only 292 days in total.
The missing data are obtained by interpolating the data at the adjacent moments. Among the data, Precipitation data of regional

Methods
To test our method, we compared with the optical flow method, ConvLSTM, U-net methods. Due to the limit of the computational resource, we use the sequence of 5 frames before time t to predict the following 5 frames. Then, the output 75 results are used to further predict the radar echo (Fig 3).

Optical flow method
We first employed the rainy motion v1, an optical flow model proposed by Ayzel et al (Ayzel et al., 2019), to see the 80 performance of the optical flow algorithm for tracking and extrapolating radar echo by our dataset. It performs poorly on the radar echo data when the leading time is up to 60 minutes. However, it performs better on satellite data, which is recorded every 30 minutes. We believe that the cloud layer motion is dominated by air advection transportation, thus the optical flow method can better simulate its motion characteristics. Also, the larger intervals between two frames make it predict better because it extrapolates fewer frames for the satellite data than those for the radar echo data at the same lead time. Convolution 85 in deep learning models not only learn the decay and initiation of radar echo, but also smooth the characteristics, which would increase its level through recursive application. Therefore, to ease the smoothing of radar echo we decide to use the results of predicted satellite data by optical flow in our multi-input models to save characteristics of precipitating system. https://doi.org/10.5194/gmd-2020-363 Preprint. Discussion started: 21 December 2020 c Author(s) 2020. CC BY 4.0 License.

ConvLSTM
ConvLSTM (Shi et al., n.d.) was one of the most classic models for the precipitation nowcasting problem. Hence, we compare 90 our model with ConvLSTM to see whether the model with multi-source data performs well when we simply formulate precipitation nowcasting as an image-to-image problem rather than spatio-temporal sequence problem (Eq. 1).
Tensor represents the radar echo map in the shape of 256×256 at time t, and tensor ̃+ 1 represents the model prediction result. 95

Multi-source Data Model (MSDM)
The model which we designed is a modified U-net model (Fig 4). Each downsample block in the encoder consists of Conv2D, Batchnorm, Leaky ReLU. Each upsample block in the decoder is Transposed Conv, Batchnorm, Dropout of 0.5 (applied to the first block), ReLU. As in U-net, there are skip connections between the encoder and decoder. We use the radar and satellite data as inputs, the output is the intensity of radar echo after half-hour (Eq. 2). The two kinds of data were fed into the encoder, 100 then they were concatenated by skip connections and flow into the decoder and transposed convolution layer (Fig 4).
Our model wants to use the weather radar echo data and Himawari 8 satellite brightness temperature data to predict the radar echo map at time t+5. Then we combined ̃+ 5 from our model and the predictions of ̃+ 5 from Optical flow for further prediction. During preprocessing, the weather radar data and Himawari 8 satellite brightness temperature data are extracted 105 which cover the area of 23.0-35.8°N, 113.0-125.8 °E with 256×256. Then the value of these data are transformed into pixels by Eq. 3 In order to improve the quality of images, we apply a modified structural similarity index (SSIM) (Wang et al., 2004) as the loss function, which is helpful to solve the blurry image problems. The loss function for the predicted image and ground truth 110 is defined as Eq. 4: In which is the predicted image, is the ground truth, and are the average value of and , respectively. 2 and 2 are variance of and , respectively. is the cross-correlation of and . 1 and 2 are small positive constants. In each calculation, a window of 3×3 is taken from the image, and then the 115 window is continuously sliding for calculation, and finally, the average value is taken as the global SSIM.

REE 120
To test if the multi-source data can help to improve REE and QPN tasks. The MSDM is trained with our dataset on Google Colab pro with Tensorflow-GPU-2.2.0 and executed on NVIDIA Tesla P100 GPU (16GB). 240 days of data are used for training, 26 days for validating and 26 days for testing. All models are compiled with Adam optimizer and the learning rate is https://doi.org/10.5194/gmd-2020-363 Preprint. Discussion started: 21 December 2020 c Author(s) 2020. CC BY 4.0 License. set at 0.001. To avoid overfitting, we apply the early-stopping strategy to monitor the loss of validation set. The critical success index (CSI= hits hits+misses+falsealarms ) is used to show the performance of different models. Other similar scores do the same work, 125 so we do not take them.
In the region we select over east China, the radar echo as well as precipitating cloud changes little between two adjacent frames (6 minutes). Therefore, the results of all models are shown every 30 minutes (Fig 5). The input of Optical flow and ConvLSTM is a sequence of 5 frames before time 0, the output is a sequence of 5 frames in the following half-hour. The input of U-net is a single frame of the radar echo data at time 0, and the input of MSDM includes a frame of satellite data and a frame of the 130 radar echo data. When the output of the first 30 minutes is got, we take it as the input to replace the real data for further prediction. After the first step prediction, the satellite data are input into the MSDM to predict QPN by the Optical flow.
Because the movement of the cloud was dominated by the advective motion, the Optical flow method is used. In Fig 5, the radar echoes predicted by the ConvLSTM, U-net and MSDM decay in the following 2 hours, while the radar echoes predicted by the Optical flow method always keep. Thus, the Optical flow method could perfectly predict the edge and shape of radar echo, which is the reason why it gets the highest CSI score with the threshold of 0.1dBZ (Table 1) on the testing set. However, the fatal weakness of the Optical flow method is that the predicted radar echo intensity is larger than the observed 140 one, which leads to the lowest CSI score with the threshold of 40 dBZ (Table 2). Besides, it cannot extrapolate the tail of radar https://doi.org/10.5194/gmd-2020-363 Preprint. Discussion started: 21 December 2020 c Author(s) 2020. CC BY 4.0 License. echo because it tracks features by the corner detector (Ayzel et al., 2019). We notice that the ConvLSTM performs the best for the strong echoes, but it cannot maintain the shape of echo. Also, there exists a phenomenon that the strong echo area is increasing solely, while the weak echo area is continuously decreasing, which is contradictory according to the fluid continuity theory. The ConvLSTM captures the temporal features from previous frames, which strengthens the intensity, but it could not 145 properly predict the initiation and decay of the whole system. That could explain why it gets the highest CSI scores with a threshold of 40 dBZ, but it looks quite different from ground truth after 60 minutes.
The ConvLSTM is prone to error accumulation due to the iterative training and needs massive computing resources. So we decide to use CNN as a substitute to treat REE as an image-to-image problem. U-net along with our MSDM could generally simulate the motion of the radar echo with maintaining its outline, but the MSDM with the satellite data could avoid the radar 150 echo decaying through iterations. Our MSDM ranks second with a threshold of 40dBZ (Table 2) and performs relatively well in the first hour with a threshold of 0.1dBZ (Table 1). We believe it keeps the merits of the Optical flow method which can maintain the pattern of the radar echo, as well as the ability to predicting the strong echo area from U-net. When the lead time is longer than 60 minutes, the MSDM performs poorly, because the accumulative error from two kinds of data was larger than either of both. Besides, the satellite data may provide more details that the radar echo may not contain, for example, data over 155 the sea, instead, these details may be treated as noise or false alarm, so the CSI scores will be lower.  We calculate the mean absolute error (MAE) and root mean squared error (RMSE) between the predicted radar echoes of four models and ground truth on the testing set, respectively ( Figure 6). the Optical flow model performs better than other models, while the U-net and MSDM model performs badly. We believe when the SSIM is used as the loss function, the CNN would 165 focus on the local features and will lead to the bad performance of the global evaluation index like MAE and RMSE. Meanwhile, we notice that the ConvLSTM produces bigger errors in the first frame of each sequence than other models. This can result from the deficiency of LSTM that cannot handle accumulative error which is magnified by the way of iterative prediction.

QPN
Previous works seem to pay little attention to QPN after they get good performance on REE tasks. Researchers tend to use an 170 empirical formula to calculate the precipitation rate based on the prediction of radar echo from models. Shi et al. (Shi et al., n.d.) employed the Z-R relationship ( = 10 + 10 ) to calculate the rainfall, here Z represents the radar echo in dBZ and R represents rainfall rate in mm/h, a and b are two constants that are calculated based on the statistical data of specific regions. We believe that this empirical formulation cannot describe the non-linear relationship between the radar echo intensity and the rainfall rate. Therefore, the random forest regression techniques from machine learning are used to describe this 175 relationship. The weather radar data and precipitation data one hour before the prediction time are used for training. The method we take is as follows. Firstly, looking for an automatic station. Then, the radar and satellite data on these grid points as well as the corresponding rainfall rate from site points are applied to train the random forest. Finally, the learned non-linear relationship is used to predict the rainfall rate an hour later.  Figure 7 shows the results of the Z-R relationship and Random forest. Since the precipitation data onto the grid points are obtained by the interpolation and might have errors, so the quantitative comparison did not make. However, this example could show that the Z-R relationship tends to overestimate the rain intensity. Figure 8 shows CSI of 480 QPN samples using different 190 methods and data. When only using radar data as the input its performance is poor. Because there is no precipitation in most of the areas, the Random forest may overfit and predict less rain. However, when we add the satellite data as input, the Random forest presents its superiority in the QPN task. Especially for the samples in the red frames in Fig 8. Hence, we believe multisource data can make the results more precise.

Conclusions and discussions 195
In order to predict QPN by machine learning based on the observed precipitation, the radar echo data, and Himawari 8 satellite brightness temperature data, we designed an image-to-image MSDM that uses the weather radar data and satellite data to predict the radar echo 30 minutes later. It performs well in the first hours with the threshold of 0.1 dBZ and ranks second with a threshold of 40dBZ within 2 hours. The MSDM combines the merits from the Optical flow method and CNN, maintains the pattern of the radar echo, and predicts their initiation and decay. The results predicted by the MSDM also contains more details 200 that U-net cannot produce. The ConvLSTM gets high scores for the strong radar echo, but it overestimates the strong echo and underestimates the weak echo. In conclusion, it shows potential in predicting areas of both strong and weak radar echo. We make an experiment by using the random forest for QPN, which gets relatively better results than that by the Z-R relationship.
It proves that the empirical formula is not suitable for all areas. So we believe that by the combination of multi-source data the radar echoes predicted by MSDM could provide more details and have fewer errors than those by the single observing data. 205 In this paper, we did not make any quality control for these data through training. Thus，the trained MSDM is more robust in the real case where there are missing data or noises. For REE task, we combined the Optical flow with Deep learning, in the future, there should be more work on the combination of multi-source data and RNNs. As for QPN, we make a trial on Random forest to estimate the precipitation. In this field, CNN should be considered for this task. Now there still exist methods to estimate precipitation rate more precisely. For example, Wu et al (Wu et al., 2020). use Graph 210 Convolutional Regression Network to produce more spatial characteristics of precipitation. For future works, we believe the predictions could be more accurate with RNNs and GRUs. Also, the precipitation rate should consider the influence of terrain and different scales. In fact, we are going to make experiments on these factors.