Articles | Volume 19, issue 21
https://doi.org/10.5194/gmd-19-10651-2026
© Author(s) 2026. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/gmd-19-10651-2026
© Author(s) 2026. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Evaluating extreme precipitation forecasts: a threshold-weighted, spatial verification approach for comparing an AI weather prediction model against a high-resolution NWP model
Nicholas Loveday
CORRESPONDING AUTHOR
Research, Science and Innovation Group, Bureau of Meteorology, Melbourne, Australia
Tracy Hertneky
NSF National Center for Atmospheric Research (NCAR) and Developmental Testbed Center (DTC), Boulder, CO, USA
Cited articles
Abdi, D., Jankov, I., Madden, P., Vargas, V., Smith, T. A., Frolov, S., Flora, M., and Potvin, C.: HRRRCast: a data-driven emulator for regional weather forecasting at convection allowing scales, arXiv [preprint], arXiv:2507.05658, 2025. a
Adamov, S., Oskarsson, J., Denby, L., Landelius, T., Hintz, K., Christiansen, S., Schicker, I., Osuna, C., Lindsten, F., Fuhrer, O., and Schemm, S.: Building Machine Learning Limited Area Models: Kilometer-Scale Weather Forecasting in Realistic Settings, arXiv [preprint], arXiv:2504.09340, 2025. a
Allen, S.: Weighted scoringRules: Emphasizing Particular Outcomes When Evaluating Probabilistic Forecasts, J. Stat. Softw., 110, https://doi.org/10.18637/jss.v110.i08, 2024. a
Allen, S., Ginsbourger, D., and Ziegel, J.: Evaluating Forecasts for High-Impact Events Using Transformed Kernel Scores, SIAM/ASA Journal on Uncertainty Quantification, 11, 906–940, https://doi.org/10.1137/22m1532184, 2023. a, b, c, d
Ayer, M., Brunk, H. D., Ewing, G. M., Reid, W. T., and Silverman, E.: An Empirical Distribution Function for Sampling with Incomplete Information, Ann. Math. Stat., 26, 641–647, https://doi.org/10.1214/aoms/1177728423, 1955. a
Baringhaus, L. and Franz, C.: On a new multivariate two-sample test, J. Multivariate Anal., 88, 190–206, https://doi.org/10.1016/s0047-259x(03)00079-4, 2004. a
Ben Bouallègue, Z., Clare, M. C. A., Magnusson, L., Gascón, E., Maier-Gerber, M., Janoušek, M., Rodwell, M., Pinault, F., Dramsch, J. S., Lang, S. T. K., Raoult, B., Rabier, F., Chevallier, M., Sandu, I., Dueben, P., Chantry, M., and Pappenberger, F.: The Rise of Data-Driven Weather Forecasting: A First Statistical Assessment of Machine Learning–Based Weather Forecasts in an Operational-Like Context, B. Am. Meteorol. Soc., 105, E864–E883, https://doi.org/10.1175/bams-d-23-0162.1, 2024. a, b
Ben-Bouallègue, Z., Prieto-Nemesio, A., Iza Wong, A., Pinault, F., van der Schee, M., and Modigliani, U.: SEEPS4ALL: an open dataset for the verification of daily precipitation forecasts using station climate statistics, Earth Syst. Sci. Data, 18, 713–720, https://doi.org/10.5194/essd-18-713-2026, 2026. a
Bi, K., Xie, L., Zhang, H., Chen, X., Gu, X., and Tian, Q.: Accurate medium-range global weather forecasting with 3D neural networks, Nature, 619, 533–538, https://doi.org/10.1038/s41586-023-06185-3, 2023. a
Brenowitz, N. D., Cohen, Y., Pathak, J., Mahesh, A., Bonev, B., Kurth, T., Durran, D. R., Harrington, P., and Pritchard, M. S.: A Practical Probabilistic Benchmark for AI Weather Models, Geophys. Res. Lett., 52, https://doi.org/10.1029/2024gl113656, 2025. a
Brier, G. W.: VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY, Mon. Weather Rev., 78, 1–3, https://doi.org/10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2, 1950. a
Bröcker, J.: Estimating reliability and resolution of probability forecasts through decomposition of the empirical score, Clim. Dynam., 39, 655–667, https://doi.org/10.1007/s00382-011-1191-1, 2011. a
Charlton-Perez, A. J., Dacre, H. F., Driscoll, S., Gray, S. L., Harvey, B., Harvey, N. J., Hunt, K. M. R., Lee, R. W., Swaminathan, R., Vandaele, R., and Volonté, A.: Do AI models produce better weather forecasts than physics-based models? A quantitative evaluation case study of Storm Ciarán, npj Clim. Atmos. Sci., 7, https://doi.org/10.1038/s41612-024-00638-w, 2024. a
Diebold, F. X. and Mariano, R. S.: Comparing Predictive Accuracy, J. Bus. Econ. Stat., 13, 253–263, https://doi.org/10.1080/07350015.1995.10524599, 1995. a
Dorninger, M., Gilleland, E., Casati, B., Mittermaier, M. P., Ebert, E. E., Brown, B. G., and Wilson, L. J.: The Setup of the MesoVICT Project, B. Am. Meteorol. Soc., 99, 1887–1906, https://doi.org/10.1175/bams-d-17-0164.1, 2018. a
Dowell, D. C., Alexander, C. R., James, E. P., Weygandt, S. S., Benjamin, S. G., Manikin, G. S., Blake, B. T., Brown, J. M., Olson, J. B., Hu, M., Smirnova, T. G., Ladwig, T., Kenyon, J. S., Ahmadov, R., Turner, D. D., Duda, J. D., and Alcott, T. I.: The High-Resolution Rapid Refresh (HRRR): An Hourly Updating Convection-Allowing Forecast Model. Part I: Motivation and System Description, Weather Forecast., 37, 1371–1395, https://doi.org/10.1175/waf-d-21-0151.1, 2022. a
Ebert, E. E.: Fuzzy verification of high-resolution gridded forecasts: a review and proposed framework, Meteorol. Appl., 15, 51–64, https://doi.org/10.1002/met.25, 2008. a
Ebert, E. E.: Neighborhood Verification: A Strategy for Rewarding Close Forecasts, Weather Forecast., 24, 1498–1510, https://doi.org/10.1175/2009waf2222251.1, 2009. a
Epstein, E. S.: A Scoring System for Probability Forecasts of Ranked Categories, J. Appl. Meteorol., 8, 985–987, https://doi.org/10.1175/1520-0450(1969)008<0985:assfpf>2.0.co;2, 1969. a
Ferro, C. A. T.: Fair scores for ensemble forecasts: Fair Scores for Ensemble Forecasts, Q. J. Roy. Meteor. Soc., 140, 1917–1923, https://doi.org/10.1002/qj.2270, 2013. a
Fovell, R. G. and Gallagher, A.: An Evaluation of Surface Wind and Gust Forecasts from the High-Resolution Rapid Refresh Model, Weather Forecast., 37, 1049–1068, https://doi.org/10.1175/waf-d-21-0176.1, 2022. a
Gilleland, E., Ahijevych, D., Brown, B. G., Casati, B., and Ebert, E. E.: Intercomparison of Spatial Forecast Verification Methods, Weather Forecast., 24, 1416–1430, https://doi.org/10.1175/2009waf2222269.1, 2009. a
Gneiting, T.: Making and Evaluating Point Forecasts, J. Am. Stat. Assoc., 106, 746–762, https://doi.org/10.1198/jasa.2011.r10138, 2011. a
Gneiting, T. and Katzfuss, M.: Probabilistic Forecasting, Annu. Rev. Stat. Appl., 1, 125–151, https://doi.org/10.1146/annurev-statistics-062713-085831, 2014. a
Gneiting, T. and Raftery, A. E.: Strictly Proper Scoring Rules, Prediction, and Estimation, J. Am. Stat. Assoc., 102, 359–378, https://doi.org/10.1198/016214506000001437, 2007. a, b
Gneiting, T. and Ranjan, R.: Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring Rules, J. Bus. Econ. Stat., 29, 411–422, https://doi.org/10.1198/jbes.2010.08110, 2011. a
Gneiting, T., Biegert, T., Kraus, K., Walz, E.-M., Jordan, A. I., and Lerch, S.: Probabilistic measures afford fair comparisons of AIWP and NWP model output, arXiv [preprint], arXiv:2506.03744, 2025. a, b, c
Henzi, A., Ziegel, J. F., and Gneiting, T.: Isotonic Distributional Regression, J. Roy. Stat. Soc. B, 83, 963–993, https://doi.org/10.1111/rssb.12450, 2021. a
Hering, A. S. and Genton, M. G.: Comparing Spatial Predictions, Technometrics, 53, 414–425, https://doi.org/10.1198/tech.2011.10136, 2011. a
Hersbach, H.: Decomposition of the Continuous Ranked Probability Score for Ensemble Prediction Systems, Weather Forecast., 15, 559–570, https://doi.org/10.1175/1520-0434(2000)015<0559:dotcrp>2.0.co;2, 2000. a
Hersbach, H., Bell, B., Berrisford, P., Hirahara, S., Horányi, A., Muñoz-Sabater, J., Nicolas, J., Peubey, C., Radu, R., Schepers, D., Simmons, A., Soci, C., Abdalla, S., Abellan, X., Balsamo, G., Bechtold, P., Biavati, G., Bidlot, J., Bonavita, M., De Chiara, G., Dahlgren, P., Dee, D., Diamantakis, M., Dragani, R., Flemming, J., Forbes, R., Fuentes, M., Geer, A., Haimberger, L., Healy, S., Hogan, R. J., Hólm, E., Janisková, M., Keeley, S., Laloyaux, P., Lopez, P., Lupu, C., Radnoti, G., de Rosnay, P., Rozum, I., Vamborg, F., Villaume, S., and Thépaut, J.-N.: The ERA5 global reanalysis, Q. J. Roy. Meteor. Soc., 146, 1999–2049, https://doi.org/10.1002/qj.3803, 2020. a
Ikeda, K., Steiner, M., Pinto, J., and Alexander, C.: Evaluation of Cold-Season Precipitation Forecasts Generated by the Hourly Updating High-Resolution Rapid Refresh Model, Weather Forecast., 28, 921–939, https://doi.org/10.1175/waf-d-12-00085.1, 2013. a
Jin, W., Weyn, J., Zhao, P., Xiang, S., Bian, J., Fang, Z., Dong, H., Sun, H., Thambiratnam, K., and Zhang, Q.: WeatherReal: A Benchmark Based on In-Situ Observations for Evaluating Weather Models, arXiv [preprint], arXiv:2409.09371, 2024. a
Keisler, R.: Forecasting Global Weather with Graph Neural Networks, arXiv [preprint], arXiv:2202.07575, 2022. a
Lam, R., Sanchez-Gonzalez, A., Willson, M., Wirnsberger, P., Fortunato, M., Alet, F., Ravuri, S., Ewalds, T., Eaton-Rosen, Z., Hu, W., Merose, A., Hoyer, S., Holland, G., Vinyals, O., Stott, J., Pritzel, A., Mohamed, S., and Battaglia, P.: Learning skillful medium-range global weather forecasting, Science, 382, 1416–1421, https://doi.org/10.1126/science.adi2336, 2023. a, b
Lang, S., Alexe, M., Chantry, M., Dramsch, J., Pinault, F., Raoult, B., Clare, M. C. A., Lessig, C., Maier-Gerber, M., Magnusson, L., Bouallègue, Z. B., Nemesio, A. P., Dueben, P. D., Brown, A., Pappenberger, F., and Rabier, F.: AIFS – ECMWF's data-driven forecasting system, arXiv [preprint], arXiv:2406.01465, 2024. a
Lang, S., Alexe, M., Clare, M. C. A., Roberts, C., Adewoyin, R., Ben Bouallègue, Z., Chantry, M., Dramsch, J., Dueben, P. D., Hahner, S., Maciel, P., Prieto-Nemesio, A., O’Brien, C., Pinault, F., Polster, J., Raoult, B., Tietsche, S., and Leutbecher, M.: AIFS-CRPS: ensemble forecasting using a model trained with a loss function based on the continuous ranked probability score, npj Artificial Intelligence, 2, https://doi.org/10.1038/s44387-026-00073-7, 2026. a
Lavers, D. A., Simmons, A., Vamborg, F., and Rodwell, M. J.: An evaluation of ERA5 precipitation for climate monitoring, Q. J. Roy. Meteor. Soc., 148, 3152–3165, https://doi.org/10.1002/qj.4351, 2022. a
Leeuwenburg, T., Loveday, N., Ebert, E. E., Cook, H., Khanarmuei, M., Taggart, R. J., Ramanathan, N., Carroll, M., Chong, S., Griffiths, A., and Sharples, J.: scores: A Python package for verifying and evaluating models and predictions with xarray, Journal of Open Source Software, 9, 6889, https://doi.org/10.21105/joss.06889, 2024. a
Leeuwenburg, T., Loveday, N., Ramanathan, N., Chong, S., Taggart, R. J., Shrestha, D., Khanarmuei, M., Cook, H., Bluett, L., Ebert, E. E., Carroll, M., Trotta, B., Sharples, J., Bishop, S., Squire, D. T., Griffiths, A., Pagano, T. C., Fisher, A., Mandelbaum, T., Jinghan, F., Smith, P. R., Abellan, E., Beunk, J., Esperson, F., Smallwood, J., and Wu, X.: scores: Metrics for the verification, evaluation and optimisation of forecasts, predictions or models, Zenodo [code], https://doi.org/10.5281/zenodo.18638494, 2026. a
Lerch, S., Thorarinsdottir, T. L., Ravazzolo, F., and Gneiting, T.: Forecaster’s Dilemma: Extreme Events and Forecast Evaluation, Stat. Sci., 32, https://doi.org/10.1214/16-sts588, 2017. a
Loveday, N.: Evaluating Extreme Precipitation Forecasts: A Threshold-Weighted, Spatial Verification Approach for Comparing an AI Weather Prediction Model Against a High-Resolution NWP Model, Zenodo [code], https://doi.org/10.5281/zenodo.20438050, 2026. a
Loveday, N. and Hertneky, T.: Data for: Evaluating Extreme Precipitation Forecasts: A Threshold-Weighted, Spatial Verification Approach for Comparing an AI Weather Prediction Model Against a High-Resolution NWP Model, Zenodo [data set], https://doi.org/10.5281/zenodo.19672887, 2026. a
Loveday, N., Griffiths, D., Leeuwenburg, T., Taggart, R. J., Pagano, T. C., Cheng, G., Plastow, K., Ebert, E. E., Templeton, C., Carroll, M., Khanarmuei, M., and Nagpal, I.: The Jive Verification System and Its Transformative Impact on Weather Forecasting Operations, B. Am. Meteorol. Soc., 105, E2047–E2063, https://doi.org/10.1175/bams-d-23-0267.1, 2024. a
Mass, C. F., Ovens, D., Westrick, K., and Colle, B. A.: Does Increasing Horizontal Resolution Produce More Skillful Forecasts?, B. Am. Meteorol. Soc., 83, 407–430, https://doi.org/10.1175/1520-0477(2002)083<0407:dihrpm>2.3.co;2, 2002. a
Matheson, J. E. and Winkler, R. L.: Scoring Rules for Continuous Probability Distributions, Management Science, 22, 1087–1096, https://doi.org/10.1287/mnsc.22.10.1087, 1976. a, b
McGovern, A., Mandelbaum, T., Rothenberg, D., Loveday, N., Potvin, C., Flora, M., Magnusson, L., Gilleland, E., and Allen, J.: Extreme Weather Bench: A framework and benchmark for evaluation of high-impact weather, arXiv [preprint], arXiv:2605.01126, 2026. a
Mittermaier, M. P.: A Strategy for Verifying Near-Convection-Resolving Model Forecasts at Observing Sites, Weather Forecast., 29, 185–204, https://doi.org/10.1175/waf-d-12-00075.1, 2014. a
Mittermaier, M. P. and Csima, G.: Ensemble versus Deterministic Performance at the Kilometer Scale, Weather Forecast., 32, 1697–1709, https://doi.org/10.1175/waf-d-16-0164.1, 2017. a
Morisseau, H., Zhu, H., Hudson, D., and de Burgh-Day, C.: Object-oriented verification of TC-Jasper rainfall forecasts: Machine learning, Tech. Rep. Bureau Research Report No. 106, Bureau of Meteorology, http://www.bom.gov.au/research/publications/researchreports/BRR-106.pdf (last access: 21 November 2025), 2025. a
Murphy, A. H. and Epstein, E. S.: Skill Scores and Correlation Coefficients in Model Verification, Mon. Weather Rev., 117, 572–582, https://doi.org/10.1175/1520-0493(1989)117<0572:ssacci>2.0.co;2, 1989. a
Nipen, T. N., Haugen, H. H., Ingstad, M. S., Nordhagen, E. M., Salihi, A. F. S., Tedesco, P., Seierstad, I. A., Kristiansen, J., Lang, S., Alexe, M., Dramsch, J., Raoult, B., Mertes, G., and Chantry, M.: Regional Data-Driven Weather Modeling with a Global Stretched Grid, Artificial Intelligence for the Earth Systems, 5, https://doi.org/10.1175/aies-d-25-0001.1, 2026. a
NWS: Automated Surface Observing System (ASOS) user's guide, Tech. rep., NOAA, 1998. a
Olivetti, L. and Messori, G.: Do data-driven models beat numerical models in forecasting weather extremes? A comparison of IFS HRES, Pangu-Weather, and GraphCast, Geosci. Model Dev., 17, 7915–7962, https://doi.org/10.5194/gmd-17-7915-2024, 2024. a
Pagano, T. C., Casati, B., Landman, S., Loveday, N., Taggart, R., Ebert, E. E., Khanarmuei, M., Jensen, T. L., Mittermaier, M., Roberts, H., Willington, S., Roberts, N., Sowko, M., Strassberg, G., Kluepfel, C., Bullock, T. A., Turner, D. D., Pappenberger, F., Osborne, N., and Noble, C.: Challenges of Operational Weather Forecast Verification and Evaluation, B. Am. Meteorol. Soc., 105, E789–E802, https://doi.org/10.1175/bams-d-22-0257.1, 2024. a
Pic, R., Dombry, C., Naveau, P., and Taillardat, M.: Proper scoring rules for multivariate probabilistic forecasts based on aggregation and transformation, Adv. Stat. Clim. Meteorol. Oceanogr., 11, 23–58, https://doi.org/10.5194/ascmo-11-23-2025, 2025. a
Price, I., Sanchez-Gonzalez, A., Alet, F., Andersson, T. R., El-Kadi, A., Masters, D., Ewalds, T., Stott, J., Mohamed, S., Battaglia, P., Lam, R., and Willson, M.: Probabilistic weather forecasting with machine learning, Nature, 637, 84–90, https://doi.org/10.1038/s41586-024-08252-9, 2024. a
Radford, J. T., Ebert-Uphoff, I., and Stewart, J. Q.: A Comparison of AI Weather Prediction and Numerical Weather Prediction Models for 1–7-Day Precipitation Forecasts, Weather Forecast., 40, https://doi.org/10.1175/waf-d-24-0081.1, 2025a. a, b, c
Radford, J. T., Ebert-Uphoff, I., Stewart, J. Q., Musgrave, K. D., DeMaria, R., Tourville, N., and Hilburn, K.: Accelerating Community-Wide Evaluation of AI Models for Global Weather Prediction by Facilitating Access to Model Output, B. Am. Meteorol. Soc., 106, E68–E76, https://doi.org/10.1175/bams-d-24-0057.1, 2025b. a
Rasp, S., Hoyer, S., Merose, A., Langmore, I., Battaglia, P., Russell, T., Sanchez-Gonzalez, A., Yang, V., Carver, R., Agrawal, S., Chantry, M., Ben Bouallegue, Z., Dueben, P., Bromberg, C., Sisk, J., Barrington, L., Bell, A., and Sha, F.: WeatherBench 2: A Benchmark for the Next Generation of Data-Driven Global Weather Models, J. Adv. Model. Earth Sy., 16, https://doi.org/10.1029/2023ms004019, 2024. a
Rodwell, M. J., Richardson, D. S., Hewson, T. D., and Haiden, T.: A new equitable score suitable for verifying precipitation in numerical weather prediction, Q. J. Roy. Meteor. Soc., 136, 1344–1363, https://doi.org/10.1002/qj.656, 2010. a
Schwartz, C. S. and Sobash, R. A.: Generating Probabilistic Forecasts from Convection-Allowing Ensembles Using Neighborhood Approaches: A Review and Recommendations, Mon. Weather Rev., 145, 3397–3418, https://doi.org/10.1175/mwr-d-16-0400.1, 2017. a, b
Siegert, S.: Simplifying and generalising Murphy’s Brier score decomposition, Q. J. Roy. Meteor. Soc., 143, 1178–1183, https://doi.org/10.1002/qj.2985, 2017. a
Stein, J. and Stoop, F.: Neighborhood-Based Ensemble Evaluation Using the CRPS, Mon. Weather Rev., 150, 1901–1914, https://doi.org/10.1175/mwr-d-21-0224.1, 2022. a
Subich, C., Husain, S. Z., Separovic, L., and Yang, J.: Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function, arXiv [preprint], arXiv:2501.19374, 2025. a
Székely, G. J. and Rizzo, M. L.: A new test for multivariate normality, J. Multivariate Anal., 93, 58–80, https://doi.org/10.1016/j.jmva.2003.12.002, 2005. a
Taggart, R.: Evaluation of point forecasts for extreme events using consistent scoring functions, Q. J. Roy. Meteor. Soc., 148, 306–320, https://doi.org/10.1002/qj.4206, 2021. a, b
Theis, S. E., Hense, A., and Damrath, U.: Probabilistic precipitation forecasts from a deterministic model: a pragmatic approach, Meteorol. Appl., 12, 257–268, https://doi.org/10.1017/s1350482705001763, 2005. a
Tustison, B., Harris, D., and Foufoula-Georgiou, E.: Scale issues in verification of precipitation forecasts, J. Geophys. Res.-Atmos., 106, 11 775–11 784, https://doi.org/10.1029/2001jd900066, 2001. a
Walz, E.-M., Henzi, A., Ziegel, J., and Gneiting, T.: Easy Uncertainty Quantification (EasyUQ): Generating Predictive Distributions from Single-Valued Model Output, SIAM Rev., 66, 91–122, https://doi.org/10.1137/22m1541915, 2024. a
Wessel, J. B., Ferro, C. A. T., Evans, G. R., and Kwasniok, F.: Improving probabilistic forecasts of extreme wind speeds by training statistical post-processing models with weighted scoring rules, Mon. Weather Rev., https://doi.org/10.1175/mwr-d-24-0151.1, 2025. a
Winkler, R. L. and Murphy, A. H.: “Good” Probability Assessors, J. Appl. Meteorol., 7, 751–758, https://doi.org/10.1175/1520-0450(1968)007<0751:pa>2.0.co;2, 1968. a
WMO: Guide to Hydrological Practices, Tech. Rep. 168, World Meteorological Organization, 1994. a
Zhang, Z., Fischer, E., Zscheischler, J., and Engelke, S.: Physics-based models outperform AI weather forecasts of record-breaking extremes, Science Advances, 12, https://doi.org/10.1126/sciadv.aec1433, 2026. a
Short summary
This study introduces a verification method that accounts for differences in grid resolution when evaluating extreme event forecasts. We apply it to an artificial intelligence-based weather prediction model and a high-resolution numerical weather prediction model. Results show that, when assessed on equivalent neighborhood scales, the high resolution numerical weather prediction model only outperforms the AI system for short lead times in predicting extreme precipitation.
This study introduces a verification method that accounts for differences in grid resolution...
Altmetrics
Final-revised paper
Preprint