Articles | Volume 16, issue 13
https://doi.org/10.5194/gmd-16-3927-2023
© Author(s) 2023. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/gmd-16-3927-2023
© Author(s) 2023. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Methods for assessment of models
| 
13 Jul 2023
Methods for assessment of models |
| 13 Jul 2023
| 13 Jul 2023
Evaluating precipitation distributions at regional scales: a benchmarking framework and application to CMIP5 and 6 models
Min-Seop Ahn
CORRESPONDING AUTHOR
PCMDI, Lawrence Livermore National Laboratory, Livermore, CA, USA
GMAO, NASA Goddard Space Flight Center, Greenbelt, MD, USA
ESSIC, University of Maryland, College Park, MD, USA
Paul A. Ullrich
PCMDI, Lawrence Livermore National Laboratory, Livermore, CA, USA
Department of Land, Air and Water Resources, University of California, Davis, CA, USA
Peter J. Gleckler
PCMDI, Lawrence Livermore National Laboratory, Livermore, CA, USA
PCMDI, Lawrence Livermore National Laboratory, Livermore, CA, USA
Ana C. Ordonez
PCMDI, Lawrence Livermore National Laboratory, Livermore, CA, USA
Angeline G. Pendergrass
Earth and Atmospheric Science, Cornell University, Ithaca, NY, USA
CGD, National Center for Atmospheric Research, Boulder, CO, USA
Related authors
No articles found.
Jung Choi, Jiwoo Lee, Kristin Chang, Paul A. Ullrich, Peter J. Gleckler, and Sang-Yoon Jun
EGUsphere, https://doi.org/10.5194/egusphere-2026-958, https://doi.org/10.5194/egusphere-2026-958, 2026
This preprint is open for discussion and under review for Geoscientific Model Development (GMD).
Short summary
Short summary
As climate risks grow, society needs reliable predictions for the coming years and decades. We developed a framework to collectively compare climate prediction systems and examine their performances on global temperature, rainfall, and sea ice. As a complementary to traditional analyses, our new framework offers tracking evolution of model performance in simulation time, helping scientists and stakeholders better understand strengths and limits of decadal climate prediction.
Hongyu Chen, Paul A. Ullrich, Julian Panetta, David Marsico, Moritz Hanke, Rajeev Jain, Chengzhu Zhang, and Robert L. Jacob
EGUsphere, https://doi.org/10.5194/egusphere-2026-636, https://doi.org/10.5194/egusphere-2026-636, 2026
This preprint is open for discussion and under review for Geoscientific Model Development (GMD).
Short summary
Short summary
Accurate data transfer between different grids is essential in climate and weather modeling. This study analyzes the geometric operations that underpin such transfers on the spherical Earth, identifies gaps and limitations in current methods, and introduces improved algorithms to ensure accuracy, reliability, and performance for next-generation modeling and data analysis systems.
Jishi Zhang, Jean–Christophe Golaz, Matthew Vincent Signorotti, Hsiang–He Lee, Peter Bogenschutz, Minda Monteagudo, Paul Aaron Ullrich, Robert S. Arthur, Stephen Po–Chedley, Philip Cameron–smith, and Jean–Paul Watson
EGUsphere, https://doi.org/10.5194/egusphere-2025-3947, https://doi.org/10.5194/egusphere-2025-3947, 2025
Short summary
Short summary
We ran a convection-permitting model with regional mesh refinement (3.25 km and 800 m) to simulate present-day wind and solar capacity factors over California, coupling it to an energy generation model. The high-resolution models captured realistic seasonal and diurnal cycles, with wind markedly better than a 25 km model and solar outperforming a 3 km operational forecast. We highlight the critical role of resolution, modeling assumptions, and data reliability in renewable energy assessment.
Forrest M. Hoffman, Birgit Hassler, Ranjini Swaminathan, Jared Lewis, Bouwe Andela, Nathaniel Collier, Dóra Hegedűs, Jiwoo Lee, Charlotte Pascoe, Mika Pflüger, Martina Stockhause, Paul Ullrich, Min Xu, Lisa Bock, Felicity Chun, Bettina K. Gier, Douglas I. Kelley, Axel Lauer, Julien Lenhardt, Manuel Schlund, Mohanan G. Sreeush, Katja Weigel, Ed Blockley, Rebecca Beadling, Romain Beucher, Demiso D. Dugassa, Valerio Lembo, Jianhua Lu, Swen Brands, Jerry Tjiputra, Elizaveta Malinina, Brian Mederios, Enrico Scoccimarro, Jeremy Walton, Philip Kershaw, André L. Marquez, Malcolm J. Roberts, Eleanor O’Rourke, Elisabeth Dingley, Briony Turner, Helene Hewitt, and John P. Dunne
EGUsphere, https://doi.org/10.5194/egusphere-2025-2685, https://doi.org/10.5194/egusphere-2025-2685, 2025
Short summary
Short summary
As Earth system models become more complex, rapid and comprehensive evaluation through comparison with observational data is necessary. The upcoming Assessment Fast Track for the Seventh Phase of the Coupled Model Intercomparison Project (CMIP7) will require fast analysis. This paper describes a new Rapid Evaluation Framework (REF) that was developed for the Assessment Fast Track that will be run at the Earth System Grid Federation (ESGF) to inform the community about the performance of models.
Katherine M. Smith, Alice M. Barthel, LeAnn M. Conlon, Luke P. Van Roekel, Anthony Bartoletti, Jean-Christophe Golaz, Chengzhu Zhang, Carolyn Branecky Begeman, James J. Benedict, Gautam Bisht, Yan Feng, Walter Hannah, Bryce E. Harrop, Nicole Jeffery, Wuyin Lin, Po-Lun Ma, Mathew E. Maltrud, Mark R. Petersen, Balwinder Singh, Qi Tang, Teklu Tesfa, Jonathan D. Wolfe, Shaocheng Xie, Xue Zheng, Karthik Balaguru, Oluwayemi Garuba, Peter Gleckler, Aixue Hu, Jiwoo Lee, Ben Moore-Maley, and Ana C. Ordoñez
Geosci. Model Dev., 18, 1613–1633, https://doi.org/10.5194/gmd-18-1613-2025, https://doi.org/10.5194/gmd-18-1613-2025, 2025
Short summary
Short summary
Version 2.1 of the U.S. Department of Energy's Energy Exascale Earth System Model (E3SM) adds the Fox-Kemper et al. (2011) mixed-layer eddy parameterization, which restratifies the ocean surface layer through an overturning streamfunction. Results include surface layer bias reduction in temperature, salinity, and sea ice extent in the North Atlantic; a small strengthening of the Atlantic meridional overturning circulation; and improvements to many atmospheric climatological variables.
Malcolm J. Roberts, Kevin A. Reed, Qing Bao, Joseph J. Barsugli, Suzana J. Camargo, Louis-Philippe Caron, Ping Chang, Cheng-Ta Chen, Hannah M. Christensen, Gokhan Danabasoglu, Ivy Frenger, Neven S. Fučkar, Shabeh ul Hasson, Helene T. Hewitt, Huanping Huang, Daehyun Kim, Chihiro Kodama, Michael Lai, Lai-Yung Ruby Leung, Ryo Mizuta, Paulo Nobre, Pablo Ortega, Dominique Paquin, Christopher D. Roberts, Enrico Scoccimarro, Jon Seddon, Anne Marie Treguier, Chia-Ying Tu, Paul A. Ullrich, Pier Luigi Vidale, Michael F. Wehner, Colin M. Zarzycki, Bosong Zhang, Wei Zhang, and Ming Zhao
Geosci. Model Dev., 18, 1307–1332, https://doi.org/10.5194/gmd-18-1307-2025, https://doi.org/10.5194/gmd-18-1307-2025, 2025
Short summary
Short summary
HighResMIP2 is a model intercomparison project focusing on high-resolution global climate models, that is, those with grid spacings of 25 km or less in the atmosphere and ocean, using simulations of decades to a century in length. We are proposing an update of our simulation protocol to make the models more applicable to key questions for climate variability and hazard in present-day and future projections and to build links with other communities to provide more robust climate information.
Bo Dong, Paul Ullrich, Jiwoo Lee, Peter Gleckler, Kristin Chang, and Travis A. O'Brien
Geosci. Model Dev., 18, 961–976, https://doi.org/10.5194/gmd-18-961-2025, https://doi.org/10.5194/gmd-18-961-2025, 2025
Short summary
Short summary
A metrics package designed for easy analysis of atmospheric river (AR) characteristics and statistics is presented. The tool is efficient for diagnosing systematic AR bias in climate models and useful for evaluating new AR characteristics in model simulations. In climate models, landfalling AR precipitation shows dry biases globally, and AR tracks are farther poleward (equatorward) in the North and South Atlantic (South Pacific and Indian Ocean).
Paul J. Durack, Karl E. Taylor, Peter J. Gleckler, Gerald A. Meehl, Bryan N. Lawrence, Curt Covey, Ronald J. Stouffer, Guillaume Levavasseur, Atef Ben-Nasser, Sebastien Denvil, Martina Stockhause, Jonathan M. Gregory, Martin Juckes, Sasha K. Ames, Fabrizio Antonio, David C. Bader, John P. Dunne, Daniel Ellis, Veronika Eyring, Sandro L. Fiore, Sylvie Joussaume, Philip Kershaw, Jean-Francois Lamarque, Michael Lautenschlager, Jiwoo Lee, Chris F. Mauzey, Matthew Mizielinski, Paola Nassisi, Alessandra Nuzzo, Eleanor O’Rourke, Jeffrey Painter, Gerald L. Potter, Sven Rodriguez, and Dean N. Williams
EGUsphere, https://doi.org/10.5194/egusphere-2024-3729, https://doi.org/10.5194/egusphere-2024-3729, 2025
Short summary
Short summary
CMIP6 was the most expansive and ambitious Model Intercomparison Project (MIP), the latest in a history, extending four decades. CMIP engaged a growing community focused on improving climate understanding, and quantifying and attributing observed climate change being experienced today. The project's profound impact is due to the combining the latest climate science and technology, enabling the latest-generation climate simulations and increasing community attention in every successive phase.
Seung H. Baek, Paul A. Ullrich, Bo Dong, and Jiwoo Lee
Geosci. Model Dev., 17, 8665–8681, https://doi.org/10.5194/gmd-17-8665-2024, https://doi.org/10.5194/gmd-17-8665-2024, 2024
Short summary
Short summary
We evaluate downscaled products by examining locally relevant co-variances during precipitation events. Common statistical downscaling techniques preserve expected co-variances during convective precipitation (a stationary phenomenon). However, they dampen future intensification of frontal precipitation (a non-stationary phenomenon) captured in global climate models and dynamical downscaling. Our study quantifies a ramification of the stationarity assumption underlying statistical downscaling.
Angeline G. Pendergrass, Michael P. Byrne, Oliver Watt-Meyer, Penelope Maher, and Mark J. Webb
Geosci. Model Dev., 17, 6365–6378, https://doi.org/10.5194/gmd-17-6365-2024, https://doi.org/10.5194/gmd-17-6365-2024, 2024
Short summary
Short summary
The width of the tropical rain belt affects many aspects of our climate, yet we do not understand what controls it. To better understand it, we present a method to change it in numerical model experiments. We show that the method works well in four different models. The behavior of the width is unexpectedly simple in some ways, such as how strong the winds are as it changes, but in other ways, it is more complicated, especially how temperature increases with carbon dioxide.
Jiwoo Lee, Peter J. Gleckler, Min-Seop Ahn, Ana Ordonez, Paul A. Ullrich, Kenneth R. Sperber, Karl E. Taylor, Yann Y. Planton, Eric Guilyardi, Paul Durack, Celine Bonfils, Mark D. Zelinka, Li-Wei Chao, Bo Dong, Charles Doutriaux, Chengzhu Zhang, Tom Vo, Jason Boutte, Michael F. Wehner, Angeline G. Pendergrass, Daehyun Kim, Zeyu Xue, Andrew T. Wittenberg, and John Krasting
Geosci. Model Dev., 17, 3919–3948, https://doi.org/10.5194/gmd-17-3919-2024, https://doi.org/10.5194/gmd-17-3919-2024, 2024
Short summary
Short summary
We introduce an open-source software, the PCMDI Metrics Package (PMP), developed for a comprehensive comparison of Earth system models (ESMs) with real-world observations. Using diverse metrics evaluating climatology, variability, and extremes simulated in thousands of simulations from the Coupled Model Intercomparison Project (CMIP), PMP aids in benchmarking model improvements across generations. PMP also enables efficient tracking of performance evolutions during ESM developments.
Justin L. Willson, Kevin A. Reed, Christiane Jablonowski, James Kent, Peter H. Lauritzen, Ramachandran Nair, Mark A. Taylor, Paul A. Ullrich, Colin M. Zarzycki, David M. Hall, Don Dazlich, Ross Heikes, Celal Konor, David Randall, Thomas Dubos, Yann Meurdesoif, Xi Chen, Lucas Harris, Christian Kühnlein, Vivian Lee, Abdessamad Qaddouri, Claude Girard, Marco Giorgetta, Daniel Reinert, Hiroaki Miura, Tomoki Ohno, and Ryuji Yoshida
Geosci. Model Dev., 17, 2493–2507, https://doi.org/10.5194/gmd-17-2493-2024, https://doi.org/10.5194/gmd-17-2493-2024, 2024
Short summary
Short summary
Accurate simulation of tropical cyclones (TCs) is essential to understanding their behavior in a changing climate. One way this is accomplished is through model intercomparison projects, where results from multiple climate models are analyzed to provide benchmark solutions for the wider climate modeling community. This study describes and analyzes the previously developed TC test case for nine climate models in an intercomparison project, providing solutions that aid in model development.
Lele Shu, Paul Ullrich, Xianhong Meng, Christopher Duffy, Hao Chen, and Zhaoguo Li
Geosci. Model Dev., 17, 497–527, https://doi.org/10.5194/gmd-17-497-2024, https://doi.org/10.5194/gmd-17-497-2024, 2024
Short summary
Short summary
Our team developed rSHUD v2.0, a toolkit that simplifies the use of the SHUD, a model simulating water movement in the environment. We demonstrated its effectiveness in two watersheds, one in the USA and one in China. The toolkit also facilitated the creation of the Global Hydrological Data Cloud, a platform for automatic data processing and model deployment, marking a significant advancement in hydrological research.
Qi Tang, Jean-Christophe Golaz, Luke P. Van Roekel, Mark A. Taylor, Wuyin Lin, Benjamin R. Hillman, Paul A. Ullrich, Andrew M. Bradley, Oksana Guba, Jonathan D. Wolfe, Tian Zhou, Kai Zhang, Xue Zheng, Yunyan Zhang, Meng Zhang, Mingxuan Wu, Hailong Wang, Cheng Tao, Balwinder Singh, Alan M. Rhoades, Yi Qin, Hong-Yi Li, Yan Feng, Yuying Zhang, Chengzhu Zhang, Charles S. Zender, Shaocheng Xie, Erika L. Roesler, Andrew F. Roberts, Azamat Mametjanov, Mathew E. Maltrud, Noel D. Keen, Robert L. Jacob, Christiane Jablonowski, Owen K. Hughes, Ryan M. Forsyth, Alan V. Di Vittorio, Peter M. Caldwell, Gautam Bisht, Renata B. McCoy, L. Ruby Leung, and David C. Bader
Geosci. Model Dev., 16, 3953–3995, https://doi.org/10.5194/gmd-16-3953-2023, https://doi.org/10.5194/gmd-16-3953-2023, 2023
Short summary
Short summary
High-resolution simulations are superior to low-resolution ones in capturing regional climate changes and climate extremes. However, uniformly reducing the grid size of a global Earth system model is too computationally expensive. We provide an overview of the fully coupled regionally refined model (RRM) of E3SMv2 and document a first-of-its-kind set of climate production simulations using RRM at an economic cost. The key to this success is our innovative hybrid time step method.
Abhishekh Kumar Srivastava, Paul Aaron Ullrich, Deeksha Rastogi, Pouya Vahmani, Andrew Jones, and Richard Grotjahn
Geosci. Model Dev., 16, 3699–3722, https://doi.org/10.5194/gmd-16-3699-2023, https://doi.org/10.5194/gmd-16-3699-2023, 2023
Short summary
Short summary
Stakeholders need high-resolution regional climate data for applications such as assessing water availability and mountain snowpack. This study examines 3 h and 24 h historical precipitation over the contiguous United States in the 12 km WRF version 4.2.1-based dynamical downscaling of the ERA5 reanalysis. WRF improves precipitation characteristics such as the annual cycle and distribution of the precipitation maxima, but it also displays regionally and seasonally varying precipitation biases.
Zeyu Xue, Paul Ullrich, and Lai-Yung Ruby Leung
Hydrol. Earth Syst. Sci., 27, 1909–1927, https://doi.org/10.5194/hess-27-1909-2023, https://doi.org/10.5194/hess-27-1909-2023, 2023
Short summary
Short summary
We examine the sensitivity and robustness of conclusions drawn from the PGW method over the NEUS by conducting multiple PGW experiments and varying the perturbation spatial scales and choice of perturbed meteorological variables to provide a guideline for this increasingly popular regional modeling method. Overall, we recommend PGW experiments be performed with perturbations to temperature or the combination of temperature and wind at the gridpoint scale, depending on the research question.
David H. Marsico and Paul A. Ullrich
Geosci. Model Dev., 16, 1537–1551, https://doi.org/10.5194/gmd-16-1537-2023, https://doi.org/10.5194/gmd-16-1537-2023, 2023
Short summary
Short summary
Climate models involve several different components, such as the atmosphere, ocean, and land models. Information needs to be exchanged, or remapped, between these models, and devising algorithms for performing this exchange is important for ensuring the accuracy of climate simulations. In this paper, we examine the efficacy of several traditional and novel approaches to remapping on the sphere and demonstrate where our approaches offer improvement.
Iris Elisabeth de Vries, Sebastian Sippel, Angeline Greene Pendergrass, and Reto Knutti
Earth Syst. Dynam., 14, 81–100, https://doi.org/10.5194/esd-14-81-2023, https://doi.org/10.5194/esd-14-81-2023, 2023
Short summary
Short summary
Precipitation change is an important consequence of climate change, but it is hard to detect and quantify. Our intuitive method yields robust and interpretable detection of forced precipitation change in three observational datasets for global mean and extreme precipitation, but the different observational datasets show different magnitudes of forced change. Assessment and reduction of uncertainties surrounding forced precipitation change are important for future projections and adaptation.
Chengzhu Zhang, Jean-Christophe Golaz, Ryan Forsyth, Tom Vo, Shaocheng Xie, Zeshawn Shaheen, Gerald L. Potter, Xylar S. Asay-Davis, Charles S. Zender, Wuyin Lin, Chih-Chieh Chen, Chris R. Terai, Salil Mahajan, Tian Zhou, Karthik Balaguru, Qi Tang, Cheng Tao, Yuying Zhang, Todd Emmenegger, Susannah Burrows, and Paul A. Ullrich
Geosci. Model Dev., 15, 9031–9056, https://doi.org/10.5194/gmd-15-9031-2022, https://doi.org/10.5194/gmd-15-9031-2022, 2022
Short summary
Short summary
Earth system model (ESM) developers run automated analysis tools on data from candidate models to inform model development. This paper introduces a new Python package, E3SM Diags, that has been developed to support ESM development and use routinely in the development of DOE's Energy Exascale Earth System Model. This tool covers a set of essential diagnostics to evaluate the mean physical climate from simulations, as well as several process-oriented and phenomenon-based evaluation diagnostics.
Vijay S. Mahadevan, Jorge E. Guerra, Xiangmin Jiao, Paul Kuberry, Yipeng Li, Paul Ullrich, David Marsico, Robert Jacob, Pavel Bochev, and Philip Jones
Geosci. Model Dev., 15, 6601–6635, https://doi.org/10.5194/gmd-15-6601-2022, https://doi.org/10.5194/gmd-15-6601-2022, 2022
Short summary
Short summary
Coupled Earth system models require transfer of field data between multiple components with varying spatial resolutions to determine the correct climate behavior. We present the Metrics for Intercomparison of Remapping Algorithms (MIRA) protocol to evaluate the accuracy, conservation properties, monotonicity, and local feature preservation of four different remapper algorithms for various unstructured mesh problems of interest. Future extensions to more practical use cases are also discussed.
Benjamin M. Sanderson, Angeline G. Pendergrass, Charles D. Koven, Florent Brient, Ben B. B. Booth, Rosie A. Fisher, and Reto Knutti
Earth Syst. Dynam., 12, 899–918, https://doi.org/10.5194/esd-12-899-2021, https://doi.org/10.5194/esd-12-899-2021, 2021
Short summary
Short summary
Emergent constraints promise a pathway to the reduction in climate projection uncertainties by exploiting ensemble relationships between observable quantities and unknown climate response parameters. This study considers the robustness of these relationships in light of biases and common simplifications that may be present in the original ensemble of climate simulations. We propose a classification scheme for constraints and a number of practical case studies.
Paul A. Ullrich, Colin M. Zarzycki, Elizabeth E. McClenny, Marielle C. Pinheiro, Alyssa M. Stansfield, and Kevin A. Reed
Geosci. Model Dev., 14, 5023–5048, https://doi.org/10.5194/gmd-14-5023-2021, https://doi.org/10.5194/gmd-14-5023-2021, 2021
Short summary
Short summary
TempestExtremes (TE) is a multifaceted framework for feature detection, tracking, and scientific analysis of regional or global Earth system datasets. Version 2.1 of TE now provides extensive support for nodal and areal features. This paper describes the algorithms that have been added to the TE framework since version 1.0 and gives several examples of how these can be combined to produce composite algorithms for evaluating and understanding atmospheric features.
Christina Heinze-Deml, Sebastian Sippel, Angeline G. Pendergrass, Flavio Lehner, and Nicolai Meinshausen
Geosci. Model Dev., 14, 4977–4999, https://doi.org/10.5194/gmd-14-4977-2021, https://doi.org/10.5194/gmd-14-4977-2021, 2021
Short summary
Short summary
Quantifying dynamical and thermodynamical components of regional precipitation change is a key challenge in climate science. We introduce a novel statistical model (Latent Linear Adjustment Autoencoder) that combines the flexibility of deep neural networks with the robustness advantages of linear regression. The method enables estimation of the contribution of a coarse-scale atmospheric circulation proxy to daily precipitation at high resolution and in a spatially coherent manner.
Cited articles
Abramowitz, G.:
Towards a public, standardized, diagnostic benchmarking system for land surface models, Geosci. Model Dev., 5, 819–827, https://doi.org/10.5194/gmd-5-819-2012, 2012.
Ahn, M. and Kang, I.:
A practical approach to scale-adaptive deep convection in a GCM by controlling the cumulus base mass flux, npj Clim. Atmos. Sci., 1, 13, https://doi.org/10.1038/s41612-018-0021-0, 2018.
Ahn, M.-S., Gleckler, P. J., Lee, J., Pendergrass, A. G., and Jakob, C.:
Benchmarking Simulated Precipitation Variability Amplitude across Time Scales, J. Climate, 35, 3173–3196, https://doi.org/10.1175/JCLI-D-21-0542.1, 2022.
Ahn, M.-S., Ullrich, P. A., Lee, J., Gleckler, P. J., Ma, H.-Y., Terai, C. R., Bogenschutz, P. A., and Ordonez, A. C.:
Bimodality in Simulated Precipitation Frequency Distributions and Its Relationship with Convective Parameterizations, npj Clim. Atmos. Sci., submitted, https://doi.org/10.21203/rs.3.rs-2874349/v1, 2023.
Ashouri, H., Hsu, K. L., Sorooshian, S., Braithwaite, D. K., Knapp, K. R., Cecil, L. D., Nelson, B. R., and Prat, O. P.:
PERSIANN-CDR: Daily precipitation climate data record from multisatellite observations for hydrological and climate studies, B. Am. Meteorol. Soc., 96, 69–83, https://doi.org/10.1175/BAMS-D-13-00068.1, 2015.
Chakravarti, I. M., Laha, R. G., and Roy, J.:
Handbook of Methods of Applied Statistics, Volume I: Techniques of Computation, Descriptive Methods, and Statistical Inference, John Wiley & Sons, Inc, 392–394, 1967.
Chen, D. and Dai, A.:
Precipitation Characteristics in the Community Atmosphere Model and Their Dependence on Model Physics and Resolution, J. Adv. Model. Earth Sy., 11, 2352–2374, https://doi.org/10.1029/2018MS001536, 2019.
Chen, D., Dai, A., and Hall, A.:
The Convective-To-Total Precipitation Ratio and the “Drizzling” Bias in Climate Models, J. Geophys. Res.-Atmos., 126, 1–17, https://doi.org/10.1029/2020JD034198, 2021.
Covey, C., Gleckler, P. J., Doutriaux, C., Williams, D. N., Dai, A., Fasullo, J., Trenberth, K., and Berg, A.:
Metrics for the Diurnal Cycle of Precipitation: Toward Routine Benchmarks for Climate Models, J. Climate, 29, 4461–4471, https://doi.org/10.1175/JCLI-D-15-0664.1, 2016.
Dai, A.:
Precipitation characteristics in eighteen coupled climate models, J. Climate, 19, 4605–4630, https://doi.org/10.1175/JCLI3884.1, 2006.
ESGF: Observations for Model Intercomparisons Project, ESGF [data set], https://esgf-node.llnl.gov/projects/obs4mips/ (last access: 1 July 2023), 2023a.
ESGF: ESGF@DOE/LLNL, ESGF [data set], https://esgf-node.llnl.gov/projects/esgf-llnl (last access: 1 July 2023), 2023b.
Eyring, V., Bony, S., Meehl, G. A., Senior, C. A., Stevens, B., Stouffer, R. J., and Taylor, K. E.:
Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization, Geosci. Model Dev., 9, 1937–1958, https://doi.org/10.5194/gmd-9-1937-2016, 2016.
Fiedler, S., Crueger, T., D'Agostino, R., Peters, K., Becker, T., Leutwyler, D., Paccini, L., Burdanowitz, J., Buehler, S. A., Cortes, A. U., Dauhut, T., Dommenget, D., Fraedrich, K., Jungandreas, L., Maher, N., Naumann, A. K., Rugenstein, M., Sakradzija, M., Schmidt, H., Sielmann, F., Stephan, C., Timmreck, C., Zhu, X., and Stevens, B.:
Simulated Tropical Precipitation Assessed across Three Major Phases of the Coupled Model Intercomparison Project (CMIP), Mon. Weather Rev., 148, 3653–3680, https://doi.org/10.1175/MWR-D-19-0404.1, 2020.
Gleckler, P., Doutriaux, C., Durack, P., Taylor, K., Zhang, Y., Williams, D., Mason, E., and Servonnat, J.:
A More Powerful Reality Test for Climate Models, Eos, 97, 20–24, https://doi.org/10.1029/2016EO051663, 2016.
Gleckler, P. J., Taylor, K. E., and Doutriaux, C.:
Performance metrics for climate models, J. Geophys. Res.-Atmos., 113, 1–20, https://doi.org/10.1029/2007JD008972, 2008.
Haarsma, R. J., Roberts, M. J., Vidale, P. L., Senior, C. A., Bellucci, A., Bao, Q., Chang, P., Corti, S., Fučkar, N. S., Guemas, V., von Hardenberg, J., Hazeleger, W., Kodama, C., Koenigk, T., Leung, L. R., Lu, J., Luo, J.-J., Mao, J., Mizielinski, M. S., Mizuta, R., Nobre, P., Satoh, M., Scoccimarro, E., Semmler, T., Small, J., and von Storch, J.-S.: High Resolution Model Intercomparison Project (HighResMIP v1.0) for CMIP6, Geosci. Model Dev., 9, 4185–4208, https://doi.org/10.5194/gmd-9-4185-2016, 2016.
Hersbach, H., Bell, B., Berrisford, P., Hirahara, S., Horányi, A., Muñoz‐Sabater, J., Nicolas, J., Peubey, C., Radu, R., Schepers, D., Simmons, A., Soci, C., Abdalla, S., Abellan, X., Balsamo, G., Bechtold, P., Biavati, G., Bidlot, J., Bonavita, M., Chiara, G., Dahlgren, P., Dee, D., Diamantakis, M., Dragani, R., Flemming, J., Forbes, R., Fuentes, M., Geer, A., Haimberger, L., Healy, S., Hogan, R., Hólm, E., Janisková, M., Keeley, S., Laloyaux, P., Lopez, P., Lupu, C., Radnoti, G., Rosnay, P., Rozum, I., Vamborg, F., Villaume, S., and Thépaut, J.:
The ERA5 global reanalysis, Q. J. Roy. Meteor. Soc., 146, 1999–2049, https://doi.org/10.1002/qj.3803, 2020.
Huffman, G. J., Adler, R. F., Morrissey, M. M., Bolvin, D. T., Curtis, S., Joyce, R., McGavock, B., and Susskind, J.:
Global Precipitation at One-Degree Daily Resolution from Multisatellite Observations, J. Hydrometeorol., 2, 36–50, https://doi.org/10.1175/1525-7541(2001)002<0036:GPAODD>2.0.CO;2, 2001.
Huffman, G. J., Bolvin, D. T., Nelkin, E. J., Wolff, D. B., Adler, R. F., Gu, G., Hong, Y., Bowman, K. P., and Stocker, E. F.:
The TRMM Multisatellite Precipitation Analysis (TMPA): Quasi-Global, Multiyear, Combined-Sensor Precipitation Estimates at Fine Scales, J. Hydrometeorol., 8, 38–55, https://doi.org/10.1175/JHM560.1, 2007.
Huffman, G. J., Bolvin, D. T., Braithwaite, D., Hsu, K.-L., Joyce, R. J., Kidd, C., Nelkin, E. J., Sorooshian, S., Stocker, E. F., Tan, J., Wolff, D. B., and Xie, P.:
Integrated Multi-satellite Retrievals for the Global Precipitation Measurement (GPM) Mission (IMERG), Adv. Glob. Change Res., 67, 343–353, 2020.
Iturbide, M., Gutiérrez, J. M., Alves, L. M., Bedia, J., Cerezo-Mota, R., Cimadevilla, E., Cofiño, A. S., Di Luca, A., Faria, S. H., Gorodetskaya, I. V., Hauser, M., Herrera, S., Hennessy, K., Hewitt, H. T., Jones, R. G., Krakovska, S., Manzanas, R., Martínez-Castro, D., Narisma, G. T., Nurhati, I. S., Pinto, I., Seneviratne, S. I., van den Hurk, B., and Vera, C. S.:
An update of IPCC climate reference regions for subcontinental analysis of climate model data: definition and aggregated datasets, Earth Syst. Sci. Data, 12, 2959–2970, https://doi.org/10.5194/essd-12-2959-2020, 2020.
Khodadoust Siuki, S., Saghafian, B., and Moazami, S.:
Comprehensive evaluation of 3-hourly TRMM and half-hourly GPM-IMERG satellite precipitation products, Int. J. Remote Sens., 38, 558–571, https://doi.org/10.1080/01431161.2016.1268735, 2017.
Kooperman, G. J., Pritchard, M. S., Burt, M. A., Branson, M. D., and Randall, D. A.:
Robust effects of cloud superparameterization on simulated daily rainfall intensity statistics across multiple versions of the Community Earth System Model, J. Adv. Model. Earth Sy., 8, 140–165, https://doi.org/10.1002/2015MS000574, 2016.
Kooperman, G. J., Pritchard, M. S., O'Brien, T. A., and Timmermans, B. W.:
Rainfall From Resolved Rather Than Parameterized Processes Better Represents the Present-Day and Climate Change Response of Moderate Rates in the Community Atmosphere Model, J. Adv. Model. Earth Sy., 10, 971–988, https://doi.org/10.1002/2017MS001188, 2018.
Lee, J., Gleckler, P., Ordonez, A., Ahn, M.-S., Ullrich, P., Vo, T., Boutte, J., Doutriaux, C., Durack, P., Shaheen, Z., Muryanto, L., Painter, J., and Krasting, J.: PCMDI/pcmdi_metrics: PMP Version 2.5.1 (v2.5.1), Zenodo [code], https://doi.org/10.5281/zenodo.7231033, 2022.
Leung, L. R., Boos, W. R., Catto, J. L., A. DeMott, C., Martin, G. M., Neelin, J. D., O’Brien, T. A., Xie, S., Feng, Z., Klingaman, N. P., Kuo, Y.-H., Lee, R. W., Martinez-Villalobos, C., Vishnu, S., Priestley, M. D. K., Tao, C., and Zhou, Y.: Exploratory Precipitation Metrics: Spatiotemporal Characteristics, Process-Oriented, and Phenomena-Based, J. Climate, 35, 3659–3686, https://doi.org/10.1175/JCLI-D-21-0590.1, 2022.
Lin, Y., Zhao, M., Ming, Y., Golaz, J.- C., Donner, L. J., Klein, S. A., Ramaswamy, V., and Xie, S.:
Precipitation Partitioning, Tropical Clouds, and Intraseasonal Variability in GFDL AM2, J. Climate, 26, 5453–5466, https://doi.org/10.1175/JCLI-D-12-00442.1, 2013.
Ma, H., Klein, S. A., Lee, J., Ahn, M., Tao, C., and Gleckler, P. J.:
Superior Daily and Sub-Daily Precipitation Statistics for Intense and Long-Lived Storms in Global Storm-Resolving Models, Geophys. Res. Lett., 49, e2021GL096759, https://doi.org/10.1029/2021GL096759, 2022.
MacQueen, J. B.:
Some methods for classification and analysis of multivariate observations, Statistical Laboratory of the University of California, Berkeley, 21 June– 18 July 1965 and 27 December 1965–7 January 1966, 5.1, 281–297, 1967.
Martinez-Villalobos, C. and Neelin, J. D.:
Why Do Precipitation Intensities Tend to Follow Gamma Distributions?, J. Atmos. Sci., 76, 3611–3631, https://doi.org/10.1175/JAS-D-18-0343.1, 2019.
Martinez-Villalobos, C., Neelin, J. D., and Pendergrass, A. G.:
Metrics for Evaluating CMIP6 Representation of Daily Precipitation Probability Distributions, J. Climate, 1–79, https://doi.org/10.1175/JCLI-D-21-0617.1, 2022.
Meehl, G. A., Boer, G. J., Covey, C., Latif, M., and Stouffer, R. J.:
The Coupled Model Intercomparison Project (CMIP), B. Am. Meteorol. Soc., 81, 313–318, https://doi.org/10.1175/1520-0477(2000)081<0313:TCMIPC>2.3.CO;2, 2000.
Meehl, G. A., Covey, C., McAvaney, B., Latif, M., and Stouffer, R. J.:
Overview of the Coupled Model Intercomparison Project, B. Am. Meteorol. Soc., 86, 89–96, 2005.
Meehl, G. A., Covey, C., Delworth, T., Latif, M., McAvaney, B., Mitchell, J. F. B., Stouffer, R. J., and Taylor, K. E.:
The WCRP CMIP3 Multimodel Dataset: A New Era in Climate Change Research, B. Am. Meteorol. Soc., 88, 1383–1394, https://doi.org/10.1175/BAMS-88-9-1383, 2007.
PCMDI: Benchmarking Simulated Precipitation, PCMDI [data set], https://pcmdi.llnl.gov/research/metrics/precip/, last access: 1 July 2023.
Pendergrass, A. G. and Deser, C.:
Climatological Characteristics of Typical Daily Precipitation, J. Climate, 30, 5985–6003, https://doi.org/10.1175/JCLI-D-16-0684.1, 2017.
Pendergrass, A. G. and Hartmann, D. L.:
Two Modes of Change of the Distribution of Rain, J. Climate, 27, 8357–8371, https://doi.org/10.1175/JCLI-D-14-00182.1, 2014.
Pendergrass, A. G. and Knutti, R.:
The Uneven Nature of Daily Precipitation and Its Change, Geophys. Res. Lett., 45, 11980–11988, https://doi.org/10.1029/2018GL080298, 2018.
Pendergrass, A. G., Gleckler, P. J., Leung, L. R., and Jakob, C.:
Benchmarking Simulated Precipitation in Earth System Models, B. Am. Meteorol. Soc., 101, E814–E816, https://doi.org/10.1175/BAMS-D-19-0318.1, 2020.
Perkins, S. E., Pitman, A. J., Holbrook, N. J., and McAneney, J.:
Evaluation of the AR4 Climate Models' Simulated Daily Maximum Temperature, Minimum Temperature, and Precipitation over Australia Using Probability Density Functions, J. Climate, 20, 4356–4376, https://doi.org/10.1175/JCLI4253.1, 2007.
Roca, R., Alexander, L. V., Potter, G., Bador, M., Jucá, R., Contractor, S., Bosilovich, M. G., and Cloché, S.:
FROGS: a daily 1∘ × 1∘ gridded precipitation database of rain gauge, satellite and reanalysis products, Earth Syst. Sci. Data, 11, 1017–1035, https://doi.org/10.5194/essd-11-1017-2019, 2019.
Satoh, M., Stevens, B., Judt, F., Khairoutdinov, M., Lin, S.-J., Putman, W. M., and Düben, P.: Global cloud resolving models, Curr Clim Change Rep., 5, 172–184, https://doi.org/10.1007/s40641-019-00131-0, 2019.
Stephens, M. A.:
EDF Statistics for Goodness of Fit and Some Comparisons, J. Am. Stat. Assoc., 69, 730–737, https://doi.org/10.2307/2286009, 1974.
Stevens, B., Satoh, M., Auger, L., Biercamp, J., Bretherton, C. S., Chen, X., Düben, P., Judt, F., Khairoutdinov, M., Klocke, D., Kodama, C., Kornblueh, L., Lin, S.-J., Neumann, P., Putman, W. M., Röber, N., Shibuya, R., Vanniere, B., Vidale, P. L., Wedi, N., and Zhou, L.: DYAMOND: the DYnamics of the Atmospheric general circulation Modeled On Non-hydrostatic Domains, Prog. Earth Planet. Sci., 6, 61, https://doi.org/10.1186/s40645-019-0304-z, 2019.
Sun, Y., Solomon, S., Dai, A., and Portmann, R. W.:
How Often Does It Rain?, J. Climate, 19, 916–934, https://doi.org/10.1175/JCLI3672.1, 2006.
Swenson, L. M. and Grotjahn, R.:
Using Self-Organizing Maps to Identify Coherent CONUS Precipitation Regions, J. Climate, 32, 7747–7761, https://doi.org/10.1175/JCLI-D-19-0352.1, 2019.
Tang, S., Gleckler, P., Xie, S., Lee, J., Ahn, M.- S., Covey, C., and Zhang, C.:
Evaluating Diurnal and Semi-Diurnal Cycle of Precipitation in CMIP6 Models Using Satellite- and Ground-Based Observations, J. Climate, 1–56, https://doi.org/10.1175/JCLI-D-20-0639.1, 2021.
Taylor, K. E., Stouffer, R. J., and Meehl, G. A.:
An overview of CMIP5 and the experiment design, B. Am. Meteorol. Soc., 93, 485–498, https://doi.org/10.1175/BAMS-D-11-00094.1, 2012.
Trenberth, K. E. and Zhang, Y.:
How Often Does It Really Rain?, B. Am. Meteorol. Soc., 99, 289–298, https://doi.org/10.1175/BAMS-D-17-0107.1, 2018.
Trenberth, K. E., Dai, A., Rasmussen, R. M., and Parsons, D. B.:
The Changing Character of Precipitation, B. Am. Meteorol. Soc., 84, 1205–1218, https://doi.org/10.1175/BAMS-84-9-1205, 2003.
Trenberth, K. E., Zhang, Y., and Gehne, M.:
Intermittency in Precipitation: Duration, Frequency, Intensity, and Amounts Using Hourly Data, J. Hydrometeorol., 18, 1393–1412, https://doi.org/10.1175/JHM-D-16-0263.1, 2017.
U.S. DOE:
Benchmarking Simulated Precipitation in Earth System Models Workshop Report, DOE/SC-0203, U.S. Department of Energy Office of Science, Biological and Environmental Research (BER) Program, Germantown, Maryland, USA, 2020.
Waliser, D., Gleckler, P. J., Ferraro, R., Taylor, K. E., Ames, S., Biard, J., Bosilovich, M. G., Brown, O., Chepfer, H., Cinquini, L., Durack, P. J., Eyring, V., Mathieu, P.-P., Lee, T., Pinnock, S., Potter, G. L., Rixen, M., Saunders, R., Schulz, J., Thépaut, J.-N., and Tuma, M.:
Observations for Model Intercomparison Project (Obs4MIPs): status for CMIP6, Geosci. Model Dev., 13, 2945–2958, https://doi.org/10.5194/gmd-13-2945-2020, 2020.
Wehner, M., Gleckler, P., and Lee, J.:
Characterization of long period return values of extreme daily temperature and precipitation in the CMIP6 models: Part 1, model evaluation, Weather and Climate Extremes, 30, 100283, https://doi.org/10.1016/j.wace.2020.100283, 2020.
Wei, G., Lü, H., Crow, W. T., Zhu, Y., Wang, J., and Su, J.:
Evaluation of Satellite-Based Precipitation Products from IMERG V04A and V03D, CMORPH and TMPA with Gauged Rainfall in Three Climatologic Zones in China, Remote Sens.-Basel, 10, 30, https://doi.org/10.3390/rs10010030, 2017.
Xie, P., Joyce, R., Wu, S., Yoo, S. H., Yarosh, Y., Sun, F., and Lin, R.:
Reprocessed, bias-corrected CMORPH global high-resolution precipitation estimates from 1998, J. Hydrometeorol., 18, 1617–1641, https://doi.org/10.1175/JHM-D-16-0168.1, 2017.
Zhang, C., Chen, X., Shao, H., Chen, S., Liu, T., Chen, C., Ding, Q., and Du, H.:
Evaluation and intercomparison of high-resolution satellite precipitation estimates-GPM, TRMM, and CMORPH in the Tianshan Mountain Area, Remote Sens.-Basel, 10, 1543, https://doi.org/10.3390/rs10101543, 2018.
Zhang, X., Alexander, L., Hegerl, G. C., Jones, P., Tank, A. K., Peterson, T. C., Trewin, B., and Zwiers, F. W.:
Indices for monitoring changes in extremes based on daily temperature and precipitation data, WIREs Clim. Change, 2, 851–870, https://doi.org/10.1002/wcc.147, 2011.
Short summary
We introduce a framework for regional-scale evaluation of simulated precipitation distributions with 62 climate reference regions and 10 metrics and apply it to evaluate CMIP5 and CMIP6 models against multiple satellite-based precipitation products. The common model biases identified in this study are mainly associated with the overestimated light precipitation and underestimated heavy precipitation. These biases persist from earlier-generation models and have been slightly improved in CMIP6.
We introduce a framework for regional-scale evaluation of simulated precipitation distributions...
