As Li et al. (2024) stated, precipitation forecast errors can be primarily manifested by three aspects: displacement, intensity, and area size. While these three aspects fall short of providing a comprehensive depiction of the spatial morphology and structure of precipitation, they offer a straightforward and quantitative assessment of forecast errors and are commonly used in PFV studies (Ebert and Gallus, 2009). Therefore, the augmentations of original precipitations in CLPFV should be accordingly set as Fig. 2 illustrated. Notably, to simulate the varying degrees of forecast errors in real-world precipitation forecasting, we apply multi augmentations with different magnitudes to the original precipitations, rather than only using a single augmentation.

Given that the precipitation field data are raster data, any precipitation field can be described as a set of its contained precipitation grids xk. Then Eq. (1) presents the matrix expression of xk, where i is the row number of xk, j is the column number of xk, and p is the precipitation intensity value of xk. In this way, the precipitation augmentation process can be expressed as a transformation operation performed on xk with a matrix M. All transformed grids xk′ form the augmented precipitation. By controlling the values of its elements, transformation matrix M can simulate any specified degree of three kinds of precipitation errors, laying the foundation for conducting comprehensive experiments with diverse scenarios of precipitation forecasting. 1xk=ijp Displacement augmentation is to mimic displacement errors of forecasted precipitations. Transformation matrix M here is expressed by Eq. (2), where di and dj represent the offsets of the precipitation grid in the latitude (row) and longitude (column) directions respectively, and displacement error can be calculated as di2+dj2. Displacement augmentation can be expressed as adding xk and M shown in Eq. (3).

2M=didj03xk′=xk+M=i+dij+djp Intensity augmentation is designed to simulate precipitation intensity errors of forecasted precipitations, where increasing or decreasing every grid's precipitation intensity value to a certain extent. Transformation matrix M here is expressed by Eq. (4), where t denotes the proportion of precipitation increase/decrease, and then intensity augmentation can be expressed as Eq. (5).

4M=100010001+t5xk′=M×xk=ijp(1+t) Precipitation area size augmentation is designed to simulate area size errors of forecasted precipitations, by scaling the precipitation region to a certain proportion. Scaling is performed around the centroid ic,jc of the precipitation grids with a scaling percentage denoted s, and it consists of three steps. First, move the precipitation region until its centroid arrives at the coordinate origin 0,0. Next, scale the moved precipitation region. Finally, move the scaled precipitation region back until its centroid returns to ic,jc. Hence, there are three transformation matrixes (i.e., M1, M2, and M3 shown in Eq. 6) used here, and the scaling transformed grid xk′ can be expressed by Eq. (7). Since scaling will cause changes in the number and positions of precipitation grids, it is necessary to interpolate the scaled grids. Specifically, when the precipitation is enlarged, gaps will appear between the scaled grids, and it will be filled with bilinear interpolation. When the precipitation is reduced, there will be multiple scaled grids at the same coordinate, and we average their precipitation intensities to obtain the final value of that coordinate.

6M1=-ic-jc0,M2=s000s0001,M3=icjc07xks=(M2×(xk+M1))+M3=si-ic+icsj-jc+jcp These augmented precipitations, along with the original ones, form positive sample pairs, while different original precipitations forming negative sample pairs. Together, these positive and negative sample pairs provide the training data for the next stage of features extraction. By precipitations augmentation and features extraction, CLPFV can train models using original precipitations samples even without artificial labels, and finally conduct automated precipitation forecast verification.

In the stage of features extraction, an encoder is constructed to convert precipitations data (in raster format) into high-level abstract features that capture key patterns, which enables the next stage to calculate verification scores by quantifying the feature-based similarity between forecasted and observed precipitations. Two components are critical in this encoder: the backbone network, which converts raster data into feature vectors, and the loss function, which optimizes the representativeness of high-level abstract features.

In this research, we employ ResNet-18 as the backbone network because it can effectively alleviate the vanishing gradient issue and has the advantage of being lightweight (He et al., 2016). As Fig. 3 illustrated, ResNet18 starts initial features extraction of precipitation raster data via a convolutional kernel of 7×7 size and a stride of 2. Subsequently, a max pooling layer also with a stride of 2 reduces the size of the feature maps, thereby decreasing the complexity of the subsequent computations. The feature maps are then input into four residual blocks, each containing two 3×3 convolutional layers and one 1×1 convolutional layer. In these four residual blocks, the number of channels in the 3×3 convolutional layers is set to 64, 128, 256, and 512 in succession. This incremental setup facilitates the network in progressively extracting richer information about the precipitation raster data. The presence of the 1×1 convolutional layer effectively alleviates the vanishing gradient problem commonly encountered in the networks. After the residual blocks, an average pooling layer with a stride of 2 is employed to further reduce the size of the feature maps. Finally, these features are connected to a fully connected layer, generating a high-level features vector that captures of the precipitation raster data. To minimize feature redundancy and improve their representation, a projection head is added to the end of backbone network based on a multilayer perceptron structure (Chen et al., 2020), which is responsible for reducing the features' dimensionality output by the backbone network. Specifically, the projection headfirst takes the backbone network's output features vector as input, applies a linear transformation through a fully connected layer, and then uses an activation function to introduce non-linearity into the features, ultimately outputting a low-dimensional features vector.

After determining to employ ResNet-18 as the backbone network, it is necessary to use appropriate loss functions in the training process to guide the network optimization. Since conventional contrastive learning verification tasks only need to determine whether data belongs to a specific category or identify its class membership, their loss functions can just pull positive samples closer while push negative samples apart in the features space, e.g., InfoNCE (Oord et al., 2018), one of the mostly used loss functions. However, CLPFV is designed to reflect different degrees of forecast errors by its verification scores, which demands the loss function not only to distinguish between positive and negative samples but also to differentiate positive samples with varying tiers (i.e., precipitations with multi augmentations) to reflect the error degrees. To achieve this, we represent an improved loss function by connecting a penalty term with InfoNCE. As illustrated in Fig. 4, this improved loss function maintains the capability of pulling positive samples closer and pushing negative samples apart, moreover, it additionally introduces corresponding penalties based on the augmentation scale of precipitations.

Equation (8) shows CLPFV's loss function in detail, where fi is the ith original precipitation, f+ is its augmented precipitation (i.e., its positive sample), and f- is another original precipitation (i.e., its negative sample). All of them (fi, f+, and f-) are temporary extracted features of corresponding precipitations in the training process. p(f+) is the punishment function of augmented precipitations. Equation (9) shows the specific equation of punishment function p(f+), which is the combination of the magnitude square of all kinds augmentations m with a penalty coefficient λ. Since CLPFV should tolerate minor forecast errors while reflecting significant ones, the greater the magnitude of augmentation, the stronger the penalty will be. Additionally, since both intensity and area size augmentation can be set in either positive or negative directions, a quadratic function is suitable and is therefore adopted as p(f+) in this research. Specifically, the quadratic function provides a smooth and differentiable penalty for model training and features extraction. It ensures the forecast verification remains tolerant of minor errors while applying increasingly strict penalties to larger errors, effectively distinguishing different ties of forecast quality.

8LCLPV=LInfoNCE+Lpenalty=-log⁡exp⁡fi⋅f+τ∑exp⁡fi⋅f-τ+|fi⋅f+-pf+|9pf+=λ⋅(mdisplacement2+mintensity2+mareasize2) During the training process, guided by the target of minimizing the improved loss function, ResNet-18 and the extracted features are iteratively refined. This enables the final extracted high-level abstract features to be more representative of precipitations.

Since CLPFV evaluates precipitation forecasts from a holistic perspective, rather than being based on grids like existing methods (point-to-point and spatial verification methods), it means that CLPFV cannot directly use the verification scores of existing methods (e.g., TS score). Consequently, CLPFV requires a specifically designed verification score.

After stages 1 and 2, CLPFV extracts high-level feature vectors from the forecasted and observed precipitations respectively. Their similarity can ideally reflect the accuracy of precipitation forecasts: higher similarity indicates more accurate forecasts, while lower similarity suggests larger forecast errors. Therefore, the verification score of CLPFV should be directly calculated via the similarity of feature vectors.

Considering that cosine similarity can avoid the impact of features lengths and is better to capture the essential similarity, we employ it to calculate the similarity between the forecasted and observed precipitations as CLPFV's verification score. Then Cosine similarity (verification score of CLPFV) can be expressed as Eq. (10), where fF and fO here are final extracted feature vectors of forecasted and observed precipitations respectively. 10similarityfF,fO=fF⋅fOfFfO

We employed the IFS (Integrated Forecasting System) dataset (Persson and Grazzini, 2007) to train CLPFV and conduct comparative experiments, covering East China region (28 to 34.5° N, 115 to 122° E) for the period 2017 to 2021, with spatial resolution of 0.125° and temporal resolution of 3 h. This dataset contains 17622 precipitations samples. All samples are divided into two parts at a ratio of around 3 : 1 through alternating sampling, with 13 500 used for training and remaining 4122 used for testing. An example of used IFS precipitation sample is given in Fig. 5.

To conduct experts survey experiments, we used the analysis and forecast products provided by CMA (China Meteorological Administration) covering the Central-South China region (22.1 to 32.6° N, 106.25 to 116.7° E) for the period of April 2018. We specifically selected a different research area and period to conduct expert survey experiments, aiming to examine whether the contrastive learning based CLPFV possesses good generalization capability, rather thanmerely achieving high accuracy due to overfitting to the training data. The used CMA dataset contains a subset of analysis precipitation product as observed precipitations, known as CMPAS (CMA Multisource Precipitation Analysis System, Shen et al., 2018) and three subsets of forecasted precipitations from different operational numerical weather prediction models (regional short-term and mesoscale (MESO), GuangDong (GD), and ShangHai (SH)) provided by CMA. The CMA precipitation data have an identical time resolution of 3 h and were uniformly interpolated to a spatial resolution of 0.125° too. The dataset CMA consists of seven samples, each corresponding to one of the seven questions in the expert survey questionnaire. Each sample contains one observed and three forecasted precipitation products at identical timestamps. An example of the observed precipitation distribution from the CMA dataset is given in Fig. 6.

In this research, the comparison methods for CLPFV are selected as follows.

Traditional point-to-point methods with POD, FAR, and TS. These verification scores are calculated based on Hits (true positives), False Alarms (false positives), and Misses (false negatives) of the precipitation forecasting confusion matrix. The POD measures how point-to-point methods accurately identify actual rainfall events, calculated as (Hits / (Hits + Misses)). The FAR measures the forecast precision and is calculated as (False Alarms / (Hits + False Alarms)). The TS provides a more balanced measure by calculating (Hits / (Hits + Misses + False Alarms)), effectively penalizing both Misses and False alarms to give a robust score of forecasting performance.

CLPFV, POD, TS, and FSS have the same value range of [0, 1], whereas their higher values all indicate more accurate forecasts. But FAR, S, A, and L follow the opposite trend, i.e., their smaller absolute values represent more accurate forecasts. To facilitate experimental results presentation and comparative analysis, we normalized FAR, S, A, and L to the same value range of [0, 1], with higher normalized values now indicating more accurate precipitation forecasts too. The detailed introduction of normalization is provided in the Supplement.

To validate whether CLPFV can effectively verify precipitation forecast errors (i.e., being tolerant of minor errors and strict with significant ones), we need to establish gradient precipitation forecast error scenarios in comparative experiments to comprehensively cover different error degrees. Since the error degree in actual precipitation forecasts is difficult to foresee and control, directly obtaining uniformly distributed error gradients from real-world forecasted and observed precipitations is impractical. Therefore, we artificially generated the required series of gradient forecast error scenarios.

The specific comparative experimental setup is as follows. First, we use the original precipitations from the IFS test data as the simulated observed precipitations, and generate a series of simulated forecasted precipitations by applying positive and negative biases systematically to original precipitations. Table 1 presents the biases configurations for displacement, precipitation intensity, and area size in simulated forecasted precipitations.

The displacement bias represents the number of grids shifted in horizontal direction, e.g., +1 indicates a one-grid eastward shift, while -3 denotes a three-grid westward displacement shift. The intensity bias represents the multiplicative factor applied to precipitation intensity values at each grid, e.g., 0.5 reduces intensity by half, whereas 2 doubles the original values. The area size bias represents the scaling factor for the precipitation area size while maintaining a fixed centroid, e.g., 0.5 contracts the area size by 50 %, and 1.5 expands it to 150 % of the original. Figure 7 demonstrates several examples of simulated forecast precipitations with various biases configurations.

Subsequently, by inputting the simulated forecasted precipitations (i.e., generated gradient biased precipitations) and their corresponding simulated observation precipitations (i.e., the original precipitations) into verifications, we obtained the verification scores of CLPFV and comparison methods. Figure 8 presents the raw results on the IFS test data of TS, FSS-10, and CLPFV verification scores from the gradient displacement biases experiments (all raw results of comparative experiments are presented in the Supplement). According to Fig. 8, we can find that TS declined rapidly when displacement biases started to appear, reflecting its oversensitivity to minor displacement errors just as previous studies revealed. Oppositely, FSS-10 always slowly declined and thus was overly tolerant of significant displacement errors. Only CLPFV could tolerate minor errors and being strict to significant errors, thereby effectively reflecting different degrees of displacement errors.

Since directly overlaying all raw results of CLPFV and comparison methods in a single plot would make it difficult to read and analyze, we provide an optimized plot of comparative experimental results to discuss. Therefore, we conducted statistics on the raw verification scores of each verification method across different biases, calculating their average scores and standard deviations. Subsequently, by connecting the average verification scores of each bias and plotting 95 % confidence intervals, we transformed the originally dense scatters (just as Fig. 8 shows) into clear curves, making the overlay analysis of all verification methods possible. The final comparative experimental results are shown in Fig. 9. In addition, we introduced a downward-opening parabola in Fig. 9 as a benchmark reference curve for compare different PFV methods more visibly. This specific parabola was selected because its shape precisely characterizes the expected behavior of an ideal PFV method: it assigns the highest verification score in the absence of errors; permits a gradual decline for minor errors to ensure fault tolerance and avoid “double penalty” problem; and enforces an accelerated descent for larger errors to significantly penalize severe forecast failures.

According to the results of displacement biases experiments shown in Fig. 9, the verification scores of TS, POD, and normalized FAR dropped sharply from the centre, indicating that traditional point-to-point verification methods were overly sensitive to minor displacements. However, after reaching a certain point, their scores slow down considerably instead, suggesting insufficient penalties for substantial errors. This occurs because large displacements cause complete mismatches, stabilizing misses and false alarms. In contrast, spatial verification methods (FSS-5, FSS-10, normalized SAL) showed slow decreases, proving insensitivity to minor displacements. But they became overly tolerant of large displacements, with scores always above 0.8, overestimating forecast accuracy. The proposed CLPFV presented a rational curve and is more like reference parabola: maintaining tolerance for minor displacements while imposing stricter penalties for significant ones.

In the intensity biases experiments, both traditional point-to-point and spatial verification methods performed poorly, with their verification scores (TS, POD, normalized FAR, FSS-5, FSS-10, and normalized SAL) consistently close to 1. This is because they rely on binary thresholding, which loses intensity continuity. Specifically, even within 50 %–200 % intensity variation, most grids remained within original thresholds, preventing these methods from distinguishing intensity biased forecasts. Conversely, CLPFV successfully captured gradient intensity biases, with scores systematically decreasing as bias increased, demonstrating its unique threshold-free design.

In the area size biases experiments, TS, POD, and normalized FAR remained excessively sensitive to minor biases. When areas shrank, POD and TS declined sharply due to fewer Hits and Misses, while normalized FAR stayed high because False Alarms stayed relatively constant. When areas expanded, increasing False Alarms lowered TS and normalized FAR, while POD remained stable due to unchanged Hits and Misses. As to spatial verification methods, FSS-5 and FSS-10 effectively reduced oversensitivity for area size reduction but failed to penalize expansion adequately as their verification scores decline slowed. The neighbor smoothing of FSS further inflated scores, sometimes exceeding traditional point-to-point methods like TS and normalized FAR. Normalized SAL remained above 0.9, indicating ineffectiveness against area size errors. Only CLPFV maintained robust performance, tolerating minor errors while strictly penalizing large ones and assigning appropriate scores based on bias degree.

The comparison experiments indicated that CLPFV successfully met its objectives: it showed reasonable tolerance for minor errors while strictly penalizing significant ones and reflects error degrees accurately. In contrast, traditional point-to-point methods are oversensitive, while spatial methods (with suboptimal parameters) are overtolerant of major errors. This is particularly problematic in area expansion tests, where high scores may encourage selecting models with more false alarms. Such outcomes mislead decision-makers, weaken flood control planning, and waste disaster prevention resources.

Expert manual verification is still an important benchmark for assessing the reliability of PFV methods. Therefore, we also compared CLPFV and comparison verification methods with Experts verifications in this research. To quantify how closely different verification methods align with experts, it is necessary to compare subjective expert verifications with objective verification scores under a unified standard. Accordingly, we conducted experts survey experiments as follows: First, a questionnaire is used to obtain experts' rankings of the specified precipitation forecasts. Second, to achieve the comparison of PFV's verification scores with experts manual verification, we implemented CLPFV and comparison verification methods on the same precipitation forecasts to acquire their corresponding rankings. Finally, by comparing all ranks, we could find which verification method is closer to the expert manual verification.

The questionnaire consists of seven questions, corresponding to the CMA precipitations samples mentioned in Datasets. The experts' survey was independently completed by 36 experienced forecasters. As shown in Fig. 10, experts were asked to rank the forecasted precipitations after reviewing and comparing the observed and three forecasted precipitations in a question. The ranks of each question from all experts were averaged to constitute the final expert ranking benchmark for this question. To ensure the validity of ranking, we carefully selected precipitations samples with different precipitation intensity and diverse distribution characteristics, with ensuring that the performance of each forecast model varied across different questions, avoiding tasks that were too simplistic while preventing ranking inertia. Subsequently, we applied the same verification methods (i.e., TS, POD, FAR, FSS-5, FSS-10, SAL, CLPFV). Finally, the Spearman Footrule distance (Ilyas et al., 2008) is used to quantify the similarity between each verification method and the expert ranking benchmark, and the Spearman Footrule distance SFk is calculated with Eq. (10), 11SFk=∑jNQrkj-rEjNQ where k is the verification method, j is the question of questionnaire, NQ is the number of questions, rkj is j question's ranking of verification method k, and rEj is j question's ranking benchmark of experts. Accordingly, a smaller Spearman Footrule distance indicates higher consistency between the verification method and experts.

The results of experts survey experiments are presented in Table 2. As shown, CLPFV had the smallest Spearman Footrule distance, demonstrating that CLPFV was significantly more similar with expert verifications compared to other methods. This confirms that CLPFV leveraged the ability of deep learning in simulating human analysis and thereby enhancing the reliability of its verifications. Furthermore, in our experts survey experiments, we specifically selected precipitation forecast verification tasks from different regions and periods. Therefore, the good performance of CLPFV also demonstrates its good generalization capability.

TS ranked as the second-best in experts survey experiments, indicating that TS indeed provided a more comprehensive verification than POD and FAR by simultaneously accounting for Hits, Misses, and False Alarms. Moreover, the good performance of TS also suggested that traditional point-to-point method can still yield reasonable verifications, despite some situations like “double penalty” problem.

In contrast, spatial verification methods underperformed compared to traditional point-to-point verification methods. This result seems to contradict common sense, but in fact, it is not inconsistent with existing studies. In the studies on spatial verification methods, these methods and their parameters are often specifically optimized for experimental cases, thereby appearing their upper limits of verification capabilities. However, in practical applications, spatial verification methods are highly likely to lack the required expert knowledge and pre-experimentation, resulting in mismatched parameters and rules that undermine the verification accuracy. Therefore, previous studies have mostly demonstrated the excellent performance of spatial verification methods under ideal conditions, Therefore, previous studies have mostly demonstrated the excellent performance of spatial verification methods under ideal conditions, while our research highlights their limitations in practical applications: without sufficient optimization, the performance of spatial verification methods may be inferior to that of traditional point-to-point methods. This also manifests that the generalization capability and practicality of spatial verification methods still need to be improved.

To illustrate experimental results more intuitively, we selected Question 2 of questionnaire as a typical case to analyze the differences between the rankings of PFV methods and the experts ranking benchmark.

Figure 11 shows the experts' rankings for the three numerical prediction models (MESO, GD, and SH) in Question 2. As seen in Fig. 11, the observed precipitation has two key distribution characteristics: a strip-shaped region of heavy precipitation in the centre of the study area and light precipitation across the entire northwest regions. As to their forecasts, GD best captured the precipitation distribution, particularly for the heavy precipitation region. MESO overestimated the heavy precipitation, while SH underestimated both the precipitation coverage and intensity. Thus, the expert ranking of them is GD (1st) > MESO (2nd) ≫ SH (3rd).

Table 3 shows the performance of each PFV method for Question 2. Due to their dependence on threshold settings, traditional point-to-point methods (TS, POD, FAR) failed to reflect the distribution of heavy precipitation. As a result, they all incorrectly ranked MESO as the best forecast. Notably, FAR, which only considers Hits and False Alarms, even incorrectly ranked SH as the second-best forecast. As to spatial verification methods, neighborhood-based methods (FSS-5 and FSS-10) produced the same erroneous rankings. Due to their spatial smoothing operations, the underestimation of precipitation in SH and the overestimation of heavy precipitation in MESO were both mitigated, leading to their incorrect ranking of SH and MESO. Although object-based method (SAL) was slightly better than FSS-5 and FSS-10, it still incorrectly ranked MESO as the best since its focus on overall precipitation features and ignorance of inner heavy precipitation distributions. In contrast, only CLPFV fully matched the expert ranking benchmark, especially its low score for SH accurately reflected the fact that SH's forecast was significantly worse. This case analysis demonstrates that CLPFV effectively captured precipitation distribution characteristics, yielding more reliable verification results.