In recent years, there has been rapid progress in the development of artificial intelligence weather prediction (AIWP) models, sometimes referred to as data-driven models . These models are typically based on neural networks trained on reanalysis datasets. This approach contrasts with traditional numerical weather prediction (NWP) models, which rely on equations that model physical processes. One major advantage of AIWP models is that they are significantly more computationally efficient than NWP models for making predictions once they have been trained.

Most early global AIWP models are trained on ERA5 data, and therefore have a similar or coarser grid resolution of approximately 0.25°. These include deterministic models e.g., and ensemble models e.g.,. In general, these models operate at much coarser resolutions than the high-resolution physical NWP models available to many international weather forecasting centers. However, higher-resolution, limited-area AIWP models have started to emerge more recently e.g.,, and this trend is expected to continue.

As modelling centers have access to high-resolution NWP models and with the accelerating development of higher-resolution AIWP models, there is a strong need for appropriate methodologies to compare the performance of models with differing spatial resolutions. Such comparisons are important to help meteorological agencies make informed decisions regarding future modelling strategies and to assist operational meteorologists and forecast system developers in understanding each model's strengths and weaknesses.

This raises several key questions we aim to address.

First, most benchmarking efforts of global AIWP models to date have relied on point-to-point verification methods e.g.,, with perhaps the exception of , who applied an object-based verification approach. Point-to-point methods may be appropriate for evaluating forecasts at individual locations (e.g., for end-users accessing weather forecasts via mobile apps). However, meteorologists and forecast systems often use forecasts spatially, where spatial coherence and physical realism are important.

Point-to-point verification methods tend to favour smoother forecasts than are typically observed in gridded observations . When spatial structure is important, point-to-point metrics can suffer from the “double penalty” effect . This can occur when a forecast feature matches the observed shape and intensity but is slightly displaced in location, leading to it being penalised twice. It is penalised once for the false prediction at the forecast location, and once for the missed event at the observed location. This issue is particularly problematic for high-resolution models, which aim to represent fine-scale features. Lower-resolution models that fail to capture the feature at all may score better under point-to-point metrics, because they incur only a single penalty. Thus, while high-resolution NWP models may produce forecasts that appear more realistic to meteorologists, they may not perform better under standard point-to-point verification metrics .

To address the limitations of point-based verification, a wide range of spatial verification methods have been developed over the past two and a half decades . These methods are typically grouped into five categories: neighbourhood methods, scale-separation methods, feature-based (or object-based) methods, field-deformation methods, and distance measures. In this paper, we focus on an approach within the neighbourhood methods category . Neighbourhood methods are verification approaches that evaluate forecasts using values within a local neighbourhood surrounding the observation location.

Second, there is a pressing need to assess how well both AIWP and high-resolution NWP models predict extreme weather events. Much of the existing literature has focused on a limited number of case studies e.g.,. While case studies are valuable, they may be prone to selection bias or may not generalise across more cases. Focusing solely on cases where extremes occur can lead to the “forecaster's dilemma” . The forecaster's dilemma arises when forecasters or modellers must choose between issuing honest forecasts and adjusting their forecasts to optimise a particular scoring rule (we refer to the latter as “hedging”). Proper scores discourage hedging and avoid the “forecaster's dilemma”. A scoring rule is proper if the expected score is optimised by producing a forecast that corresponds to the forecaster's true belief .

There are some recent examples of broader evaluation of extremes across many events. For example, and evaluated the discrimination ability of AIWP models to differentiate between climatological extreme temperature events and non-events. assessed performance in forecasting temperature and wind extremes primarily by conditioning on observed extremes, but also included an evaluation using a threshold-weighted squared error . recently introduced a benchmark that focuses on evaluating several models to predict extremes based on a comprehensive set of events and marginal events. evaluated several AIWP models and ECMWF's HRES model against observations that are more extreme than the observations in the 1979–2017 training period of the AIWP models. Some of these studies have taken efforts to avoid, or practically reduce the impact of the “forecaster's dilemma”, although some of these approaches still suffer from this problem.

These efforts have been limited to point-to-point verification methods, and there is a need for more comprehensive verification methods that assess the performance of models predicting extremes when they are used in a user-oriented spatial framework.

The appropriate verification approach depends on how forecasts are used in practice. Weather models are used in a wide variety of ways. For example, model output could be used at each grid point to produce forecasts for a user's location (e.g., via a mobile weather application). A high-resolution ensemble could be used by an operational meteorologist to infer the likelihood of different modes of convection during the afternoon. The dynamic tropopause output of a deterministic model (or single ensemble member) can also be used by a decision-support meteorologist to construct a conceptual model of the atmosphere, aiding rapid responses to bespoke queries in an emergency response centre. These use cases require different evaluation approaches. In the first example, point-to-point evaluation using proper or consistent scoring rules may be appropriate. In the second, point-to-point verification would not capture the spatial realism of convection within the ensemble, necessitating some form of spatial processing. Similarly, in the third case, evaluating the spatial structure of the dynamic tropopause may require a spatial verification method distinct from that used in the second example. These examples illustrate that no single metric can assess all the diverse ways that weather models are used.

It is desirable that forecasts are probabilistic and evaluated using proper scoring rules

In some cases, only a single-valued forecast is required for the forecast service rather than a full predictive distribution. Consistent scoring functions can then be used to evaluate forecasts expressed through a directive in the form of a statistical functional ; for example, a 90th percentile forecast can be evaluated using a quantile loss.

This work unifies two different verification methods. We give an overview of each method.

We select HRRR and GraphCast-GFS neighbourhoods that cover similar physical areas. Because GraphCast-GFS uses a latitude–longitude grid, its physical grid spacing varies with latitude, and no single HRRR neighbourhood covers exactly the same physical area at every station. We therefore use rectangular HRRR neighbourhoods that approximate the physical areas covered by the GraphCast-GFS neighbourhoods over CONUS. This differs from most HiRA applications, which typically use square or circular neighbourhoods. Table  lists the paired neighbourhood sizes and their approximate physical dimensions.

Note that, although there is no neighbourhood size for GraphCast-GFS that is equivalent to the 3 km × 3 km point forecast for HRRR, we still evaluate the point-based HRRR to understand the impact of neighbourhood size on the scores because of the double penalty effect. The CRPS of a single ensemble member (i.e., the point-based forecast using a neighbourhood size of 1) is equivalent to the absolute loss. Likewise, threshold-weighted MAE (or threshold-weighted absolute loss) is equivalent to twCRPS for a single ensemble member.

When aggregating results spatially, we weight the results based on station density by following the approach in . This ensures that the verification results are not overly influenced by geographical regions that have a higher concentration of stations compared to others. The station density weights are allowed to vary for each timestep to account for non-continuous reporting from weather stations.

Since the CRPS is the integral of the Brier score across all thresholds (Eq. ), we decompose it to visualise the Brier scores across a range of thresholds. This provides us with greater insight within the HiRA framework as to how different models provide value at different decision thresholds. We display results for two lead times in Fig. .

Figure  shows that the mean CRPS is dominated by non-extreme precipitation amounts (i.e., mean Brier score values from lower thresholds), which is unsurprising given their higher climatological frequency. To account for this, results are split at the 30 mm threshold, and the right-hand panels use a logarithmic vertical axis to better visualise the forecast rankings.

At a 6 h lead time (top panels), HRRR had lower mean Brier scores than GraphCast-GFS for precipitation thresholds below approximately 5 mm across all neighbourhood sizes. However, at the 30 h lead time, this is only true for some neighbourhood sizes. In cases where the mean Brier score curves for HRRR and GraphCast-GFS intersect, the crossing point typically occurs at higher precipitation thresholds for larger HRRR neighbourhood sizes. This may reflect the increased likelihood of double-penalty effects at higher precipitation amounts when using point-to-point verification.

For the upper precipitation thresholds, HRRR 21 × 27 had a slightly lower mean Brier score than GraphCast-GFS 3 × 3 at the 6 h lead time and a slightly higher mean Brier score at the 30 h lead time. This is consistent with the corresponding twCRPS results in Fig. .

We now assess the agreement between the model climatology and observations to understand any model biases. Quantile–Quantile (Q–Q) plots for the 6 h and 30 h lead-time point-based forecasts and the observations are shown in Fig. . They show that the climatology of HRRR 1 × 1 closely matches the observed climatology from the ASOS observations, whereas GraphCast-GFS 1 × 1 is substantially less likely to predict heavier precipitation. This may partly reflect differences in grid resolution, with the HRRR model producing forecasts that correspond more closely to station-based observations, while GraphCast-GFS produces forecasts that may be more representative of a 0.25° grid. This is consistent with the issues involved in comparing gridded model precipitation forecasts against rain gauge observations highlighted by . To explore the relationship between grid resolution and representativeness, we also take the neighbourhood mean of HRRR 7 × 9. The mean of the HRRR 7 × 9 neighbourhood shifts the distribution away from the diagonal line, but not nearly to the same extent as GraphCast-GFS 1 × 1, suggesting that differences in the climatology of model extremes are partly driven by the smoothness of GraphCast-GFS and not just because of differences in grid resolution.

As demonstrated in the preceding section, the behaviour of GraphCast-GFS may not be consistent with in-situ observations. For this reason, it is important to understand the discrimination ability (i.e., potential predictive ability) of the models as a way of accounting for the models representing the precipitation across a grid cell rather than at a specific point. This is because meteorologists may be able to learn how the model behaves and use its output accordingly. Alternatively, a model with good discrimination ability could easily be post-processed to correct for any conditional biases. The concept of “potential” scores as a measure of potential predictive or discrimination ability has been proposed and extended multiple times, including for the CRPS e.g.,. Recently introduced a version of a potential CRPS to measure and compare the discrimination ability of a deterministic NWP model to that of an AIWP model. This was done by calibrating the model (in-sample) using IDR via the EasyUQ method to convert the single-valued forecasts to probabilistic forecasts. We adopt a related, but distinct approach for calculating the “potential” CRPS and twCRPS. Instead of using IDR on each neighbourhood member, we apply isotonic regression (in-sample) to each neighbourhood member. Isotonic regression is a method for fitting a nondecreasing function to a set of forecast-observation pairs . Specifically, we use isotonic median regression (isotonic regression under L1 loss), which targets the median functional. We adopt this approach to approximate the behaviour of an operational meteorologist who is familiar with typical model biases and applies simple calibrations to the model output. Additionally, it partly circumvents the issue in neighbourhood verification approaches where adjacent grid cells in the neighbourhood are not representative of the station site (e.g., in areas of varying topography). This approach, however, is unlikely to be as effective as IDR in producing a well-calibrated predictive distribution if neighbourhood members were used as covariates. We then calculate a potential CRPS and potential twCRPS using the forecasts calibrated via isotonic regression as shown in Figs.  and .

The potential CRPS and potential twCRPS results, which measure the discrimination ability, show that while increasing neighbourhood size leads to better discrimination ability, the HRRR and GraphCast-GFS are sometimes ranked differently compared to Figs.  and , which measure overall predictive performance rather than discrimination ability.

The potential CRPS results indicate that, when all decision thresholds are weighted equally, the HRRR 7 × 9 shows stronger discrimination ability than GraphCast-GFS 1 × 1 across all lead times, and the HRRR 21 × 27 exhibits stronger discrimination ability than GraphCast-GFS 3 × 3 at short lead times, with the discrimination ability of the two models converging thereafter.

Examining discrimination ability for extreme decision thresholds based on the potential twCRPS results, the HRRR model outperforms GraphCast-GFS at a lead time of 6 h for all equivalent neighbourhood sizes; however, this advantage diminishes with increasing lead time, and GraphCast-GFS exhibits greater discrimination ability at lead times of 24 h and beyond for the GraphCast-GFS 3 × 3 versus HRRR 21 × 27 comparison. Differences in discrimination ability for the latter comparison were not statistically significant.

This paper demonstrates an approach to evaluating how an AIWP model compares to a high-resolution physical NWP model in predicting climatologically extreme 6-hourly precipitation. It combines two existing techniques: the HiRA framework and twCRPS. Model performance was assessed within a framework representative of potential operational use by meteorologists or simple post-processing systems. As with all NWP verification methods, this approach does not definitively determine which model is superior across all use cases. The comparison includes, rather than removes, differences associated with grid spacing, effective resolution, and spatial sampling. It is not a comparison after re-gridding the forecast systems to a common spatial scale.

When models were compared using neighbourhood sizes that covered a similar physical area to generate empirical CDFs, HRRR consistently outperformed GraphCast-GFS across all lead times as measured by CRPS. However, when focusing on predictive performance of extreme precipitation, HRRR only outperformed GraphCast-GFS at short lead times.

Decomposing CRPS by decision threshold provided further insight into which thresholds each model handled more effectively. Where Brier score curves for HRRR and GraphCast-GFS intersected, this generally occurred at higher precipitation thresholds for larger HRRR neighbourhood sizes.

The approach was extended to measure the discrimination ability of the models to predict extreme precipitation. This was important to measure since GraphCast-GFS forecasts of heavy precipitation showed an under-forecast bias when evaluated against rain gauge data. Increasing the neighbourhood size led to superior discrimination ability. When GraphCast-GFS 3 × 3 versus HRRR 21 × 27 were compared, the high-resolution model had superior discrimination ability at short lead times, but not beyond 24 h.

This verification approach has several strengths: 1.

It is a spatial method that can address the double penalty effect within a specified distance.

As noted by , there is an opportunity to bridge the gap between the spatial verification methods and proper scoring rules communities. This work offers one such integration, and raises several opportunities for future research:

The approach could be extended to evaluate multivariate forecasts by using threshold-weighted multivariate scores, such as the threshold-weighted variogram score which is a proper scoring rule. This approach could be used to evaluate compound events as well as different variables simultaneously (e.g., surface wind speed, temperature and relative humidity for fire weather).

Finally, while most prior comparisons between AIWP and traditional NWP models have relied on point-to-point verification, this study shows that model rankings can differ when assessed from a spatial perspective, consistent with findings from . If AIWP models are to be operationalised, adopting spatial verification methods aligned with their practical use will be essential for accurately understanding their performance.