Articles | Volume 18, issue 4
https://doi.org/10.5194/gmd-18-961-2025
© Author(s) 2025. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/gmd-18-961-2025
© Author(s) 2025. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Methods for assessment of models
| 
20 Feb 2025
Methods for assessment of models |
| 20 Feb 2025
| 20 Feb 2025
A new metrics framework for quantifying and intercomparing atmospheric rivers in observations, reanalyses, and climate models
Bo Dong
CORRESPONDING AUTHOR
Lawrence Livermore National Laboratory, Livermore, CA, USA
Paul Ullrich
Lawrence Livermore National Laboratory, Livermore, CA, USA
Jiwoo Lee
Lawrence Livermore National Laboratory, Livermore, CA, USA
Peter Gleckler
Lawrence Livermore National Laboratory, Livermore, CA, USA
Kristin Chang
Lawrence Livermore National Laboratory, Livermore, CA, USA
Travis A. O'Brien
Department of Earth and Atmospheric Sciences, Indiana University, Bloomington, IN, USA
Climate and Ecosystem Sciences Division, Lawrence Berkeley National Lab, Berkeley, CA, USA
Related authors
Seung H. Baek, Paul A. Ullrich, Bo Dong, and Jiwoo Lee
Geosci. Model Dev., 17, 8665–8681, https://doi.org/10.5194/gmd-17-8665-2024, https://doi.org/10.5194/gmd-17-8665-2024, 2024
Short summary
Short summary
We evaluate downscaled products by examining locally relevant co-variances during precipitation events. Common statistical downscaling techniques preserve expected co-variances during convective precipitation (a stationary phenomenon). However, they dampen future intensification of frontal precipitation (a non-stationary phenomenon) captured in global climate models and dynamical downscaling. Our study quantifies a ramification of the stationarity assumption underlying statistical downscaling.
Jiwoo Lee, Peter J. Gleckler, Min-Seop Ahn, Ana Ordonez, Paul A. Ullrich, Kenneth R. Sperber, Karl E. Taylor, Yann Y. Planton, Eric Guilyardi, Paul Durack, Celine Bonfils, Mark D. Zelinka, Li-Wei Chao, Bo Dong, Charles Doutriaux, Chengzhu Zhang, Tom Vo, Jason Boutte, Michael F. Wehner, Angeline G. Pendergrass, Daehyun Kim, Zeyu Xue, Andrew T. Wittenberg, and John Krasting
Geosci. Model Dev., 17, 3919–3948, https://doi.org/10.5194/gmd-17-3919-2024, https://doi.org/10.5194/gmd-17-3919-2024, 2024
Short summary
Short summary
We introduce an open-source software, the PCMDI Metrics Package (PMP), developed for a comprehensive comparison of Earth system models (ESMs) with real-world observations. Using diverse metrics evaluating climatology, variability, and extremes simulated in thousands of simulations from the Coupled Model Intercomparison Project (CMIP), PMP aids in benchmarking model improvements across generations. PMP also enables efficient tracking of performance evolutions during ESM developments.
Ankur Mahesh, William D. Collins, Boris Bonev, Noah Brenowitz, Yair Cohen, Joshua Elms, Peter Harrington, Karthik Kashinath, Thorsten Kurth, Joshua North, Travis O'Brien, Michael Pritchard, David Pruitt, Mark Risser, Shashank Subramanian, and Jared Willard
Geosci. Model Dev., 18, 5575–5603, https://doi.org/10.5194/gmd-18-5575-2025, https://doi.org/10.5194/gmd-18-5575-2025, 2025
Short summary
Short summary
Simulating extreme weather events in a warming world is a challenging task for current weather and climate models. These models' computational cost poses a challenge in studying low-probability extreme weather. We use machine learning to construct a new probabilistic system. We give an in-depth explanation of how we constructed this system. We present a thorough pipeline to validate our method. Our method requires fewer computational resources than existing weather and climate models.
Ankur Mahesh, William D. Collins, Boris Bonev, Noah Brenowitz, Yair Cohen, Peter Harrington, Karthik Kashinath, Thorsten Kurth, Joshua North, Travis A. O'Brien, Michael Pritchard, David Pruitt, Mark Risser, Shashank Subramanian, and Jared Willard
Geosci. Model Dev., 18, 5605–5633, https://doi.org/10.5194/gmd-18-5605-2025, https://doi.org/10.5194/gmd-18-5605-2025, 2025
Short summary
Short summary
We use machine learning emulators to create a massive ensemble of simulated weather extremes. This ensemble provides a large sample size, which is essential to characterize the statistics of extreme weather events and study their physical mechanisms. Also, these ensembles can be beneficial to accurately forecast the probability of low-likelihood extreme weather.
Jishi Zhang, Jean–Christophe Golaz, Matthew Vincent Signorotti, Hsiang–He Lee, Peter Bogenschutz, Minda Monteagudo, Paul Aaron Ullrich, Robert S. Arthur, Stephen Po–Chedley, Philip Cameron–smith, and Jean–Paul Watson
EGUsphere, https://doi.org/10.5194/egusphere-2025-3947, https://doi.org/10.5194/egusphere-2025-3947, 2025
This preprint is open for discussion and under review for Geoscientific Model Development (GMD).
Short summary
Short summary
We ran a convection-permitting model with regional mesh refinement (3.25 km and 800 m) to simulate present-day wind and solar capacity factors over California, coupling it to an energy generation model. The high-resolution models captured realistic seasonal and diurnal cycles, with wind markedly better than a 25 km model and solar outperforming a 3 km operational forecast. We highlight the critical role of resolution, modeling assumptions, and data reliability in renewable energy assessment.
Forrest M. Hoffman, Birgit Hassler, Ranjini Swaminathan, Jared Lewis, Bouwe Andela, Nathaniel Collier, Dóra Hegedűs, Jiwoo Lee, Charlotte Pascoe, Mika Pflüger, Martina Stockhause, Paul Ullrich, Min Xu, Lisa Bock, Felicity Chun, Bettina K. Gier, Douglas I. Kelley, Axel Lauer, Julien Lenhardt, Manuel Schlund, Mohanan G. Sreeush, Katja Weigel, Ed Blockley, Rebecca Beadling, Romain Beucher, Demiso D. Dugassa, Valerio Lembo, Jianhua Lu, Swen Brands, Jerry Tjiputra, Elizaveta Malinina, Brian Mederios, Enrico Scoccimarro, Jeremy Walton, Philip Kershaw, André L. Marquez, Malcolm J. Roberts, Eleanor O’Rourke, Elisabeth Dingley, Briony Turner, Helene Hewitt, and John P. Dunne
EGUsphere, https://doi.org/10.5194/egusphere-2025-2685, https://doi.org/10.5194/egusphere-2025-2685, 2025
This preprint is open for discussion and under review for Geoscientific Model Development (GMD).
Short summary
Short summary
As Earth system models become more complex, rapid and comprehensive evaluation through comparison with observational data is necessary. The upcoming Assessment Fast Track for the Seventh Phase of the Coupled Model Intercomparison Project (CMIP7) will require fast analysis. This paper describes a new Rapid Evaluation Framework (REF) that was developed for the Assessment Fast Track that will be run at the Earth System Grid Federation (ESGF) to inform the community about the performance of models.
Katherine M. Smith, Alice M. Barthel, LeAnn M. Conlon, Luke P. Van Roekel, Anthony Bartoletti, Jean-Christophe Golaz, Chengzhu Zhang, Carolyn Branecky Begeman, James J. Benedict, Gautam Bisht, Yan Feng, Walter Hannah, Bryce E. Harrop, Nicole Jeffery, Wuyin Lin, Po-Lun Ma, Mathew E. Maltrud, Mark R. Petersen, Balwinder Singh, Qi Tang, Teklu Tesfa, Jonathan D. Wolfe, Shaocheng Xie, Xue Zheng, Karthik Balaguru, Oluwayemi Garuba, Peter Gleckler, Aixue Hu, Jiwoo Lee, Ben Moore-Maley, and Ana C. Ordoñez
Geosci. Model Dev., 18, 1613–1633, https://doi.org/10.5194/gmd-18-1613-2025, https://doi.org/10.5194/gmd-18-1613-2025, 2025
Short summary
Short summary
Version 2.1 of the U.S. Department of Energy's Energy Exascale Earth System Model (E3SM) adds the Fox-Kemper et al. (2011) mixed-layer eddy parameterization, which restratifies the ocean surface layer through an overturning streamfunction. Results include surface layer bias reduction in temperature, salinity, and sea ice extent in the North Atlantic; a small strengthening of the Atlantic meridional overturning circulation; and improvements to many atmospheric climatological variables.
Malcolm J. Roberts, Kevin A. Reed, Qing Bao, Joseph J. Barsugli, Suzana J. Camargo, Louis-Philippe Caron, Ping Chang, Cheng-Ta Chen, Hannah M. Christensen, Gokhan Danabasoglu, Ivy Frenger, Neven S. Fučkar, Shabeh ul Hasson, Helene T. Hewitt, Huanping Huang, Daehyun Kim, Chihiro Kodama, Michael Lai, Lai-Yung Ruby Leung, Ryo Mizuta, Paulo Nobre, Pablo Ortega, Dominique Paquin, Christopher D. Roberts, Enrico Scoccimarro, Jon Seddon, Anne Marie Treguier, Chia-Ying Tu, Paul A. Ullrich, Pier Luigi Vidale, Michael F. Wehner, Colin M. Zarzycki, Bosong Zhang, Wei Zhang, and Ming Zhao
Geosci. Model Dev., 18, 1307–1332, https://doi.org/10.5194/gmd-18-1307-2025, https://doi.org/10.5194/gmd-18-1307-2025, 2025
Short summary
Short summary
HighResMIP2 is a model intercomparison project focusing on high-resolution global climate models, that is, those with grid spacings of 25 km or less in the atmosphere and ocean, using simulations of decades to a century in length. We are proposing an update of our simulation protocol to make the models more applicable to key questions for climate variability and hazard in present-day and future projections and to build links with other communities to provide more robust climate information.
Ryan J. O'Loughlin, Dan Li, Richard Neale, and Travis A. O'Brien
Geosci. Model Dev., 18, 787–802, https://doi.org/10.5194/gmd-18-787-2025, https://doi.org/10.5194/gmd-18-787-2025, 2025
Short summary
Short summary
We draw from traditional climate modeling practices to make recommendations for machine-learning (ML)-driven climate science. Our intended audience is climate modelers who are relatively new to ML. We show how component-level understanding – obtained when scientists can link model behavior to parts within the overall model – should guide the development and evaluation of ML models. Better understanding yields a stronger basis for trust in the models. We highlight several examples to demonstrate.
Paul J. Durack, Karl E. Taylor, Peter J. Gleckler, Gerald A. Meehl, Bryan N. Lawrence, Curt Covey, Ronald J. Stouffer, Guillaume Levavasseur, Atef Ben-Nasser, Sebastien Denvil, Martina Stockhause, Jonathan M. Gregory, Martin Juckes, Sasha K. Ames, Fabrizio Antonio, David C. Bader, John P. Dunne, Daniel Ellis, Veronika Eyring, Sandro L. Fiore, Sylvie Joussaume, Philip Kershaw, Jean-Francois Lamarque, Michael Lautenschlager, Jiwoo Lee, Chris F. Mauzey, Matthew Mizielinski, Paola Nassisi, Alessandra Nuzzo, Eleanor O’Rourke, Jeffrey Painter, Gerald L. Potter, Sven Rodriguez, and Dean N. Williams
EGUsphere, https://doi.org/10.5194/egusphere-2024-3729, https://doi.org/10.5194/egusphere-2024-3729, 2025
Short summary
Short summary
CMIP6 was the most expansive and ambitious Model Intercomparison Project (MIP), the latest in a history, extending four decades. CMIP engaged a growing community focused on improving climate understanding, and quantifying and attributing observed climate change being experienced today. The project's profound impact is due to the combining the latest climate science and technology, enabling the latest-generation climate simulations and increasing community attention in every successive phase.
Seung H. Baek, Paul A. Ullrich, Bo Dong, and Jiwoo Lee
Geosci. Model Dev., 17, 8665–8681, https://doi.org/10.5194/gmd-17-8665-2024, https://doi.org/10.5194/gmd-17-8665-2024, 2024
Short summary
Short summary
We evaluate downscaled products by examining locally relevant co-variances during precipitation events. Common statistical downscaling techniques preserve expected co-variances during convective precipitation (a stationary phenomenon). However, they dampen future intensification of frontal precipitation (a non-stationary phenomenon) captured in global climate models and dynamical downscaling. Our study quantifies a ramification of the stationarity assumption underlying statistical downscaling.
Jiwoo Lee, Peter J. Gleckler, Min-Seop Ahn, Ana Ordonez, Paul A. Ullrich, Kenneth R. Sperber, Karl E. Taylor, Yann Y. Planton, Eric Guilyardi, Paul Durack, Celine Bonfils, Mark D. Zelinka, Li-Wei Chao, Bo Dong, Charles Doutriaux, Chengzhu Zhang, Tom Vo, Jason Boutte, Michael F. Wehner, Angeline G. Pendergrass, Daehyun Kim, Zeyu Xue, Andrew T. Wittenberg, and John Krasting
Geosci. Model Dev., 17, 3919–3948, https://doi.org/10.5194/gmd-17-3919-2024, https://doi.org/10.5194/gmd-17-3919-2024, 2024
Short summary
Short summary
We introduce an open-source software, the PCMDI Metrics Package (PMP), developed for a comprehensive comparison of Earth system models (ESMs) with real-world observations. Using diverse metrics evaluating climatology, variability, and extremes simulated in thousands of simulations from the Coupled Model Intercomparison Project (CMIP), PMP aids in benchmarking model improvements across generations. PMP also enables efficient tracking of performance evolutions during ESM developments.
Ankur Mahesh, Travis A. O'Brien, Burlen Loring, Abdelrahman Elbashandy, William Boos, and William D. Collins
Geosci. Model Dev., 17, 3533–3557, https://doi.org/10.5194/gmd-17-3533-2024, https://doi.org/10.5194/gmd-17-3533-2024, 2024
Short summary
Short summary
Atmospheric rivers (ARs) are extreme weather events that can alleviate drought or cause billions of US dollars in flood damage. We train convolutional neural networks (CNNs) to detect ARs with an estimate of the uncertainty. We present a framework to generalize these CNNs to a variety of datasets of past, present, and future climate. Using a simplified simulation of the Earth's atmosphere, we validate the CNNs. We explore the role of ARs in maintaining energy balance in the Earth system.
Justin L. Willson, Kevin A. Reed, Christiane Jablonowski, James Kent, Peter H. Lauritzen, Ramachandran Nair, Mark A. Taylor, Paul A. Ullrich, Colin M. Zarzycki, David M. Hall, Don Dazlich, Ross Heikes, Celal Konor, David Randall, Thomas Dubos, Yann Meurdesoif, Xi Chen, Lucas Harris, Christian Kühnlein, Vivian Lee, Abdessamad Qaddouri, Claude Girard, Marco Giorgetta, Daniel Reinert, Hiroaki Miura, Tomoki Ohno, and Ryuji Yoshida
Geosci. Model Dev., 17, 2493–2507, https://doi.org/10.5194/gmd-17-2493-2024, https://doi.org/10.5194/gmd-17-2493-2024, 2024
Short summary
Short summary
Accurate simulation of tropical cyclones (TCs) is essential to understanding their behavior in a changing climate. One way this is accomplished is through model intercomparison projects, where results from multiple climate models are analyzed to provide benchmark solutions for the wider climate modeling community. This study describes and analyzes the previously developed TC test case for nine climate models in an intercomparison project, providing solutions that aid in model development.
Lele Shu, Paul Ullrich, Xianhong Meng, Christopher Duffy, Hao Chen, and Zhaoguo Li
Geosci. Model Dev., 17, 497–527, https://doi.org/10.5194/gmd-17-497-2024, https://doi.org/10.5194/gmd-17-497-2024, 2024
Short summary
Short summary
Our team developed rSHUD v2.0, a toolkit that simplifies the use of the SHUD, a model simulating water movement in the environment. We demonstrated its effectiveness in two watersheds, one in the USA and one in China. The toolkit also facilitated the creation of the Global Hydrological Data Cloud, a platform for automatic data processing and model deployment, marking a significant advancement in hydrological research.
Arjun Babu Nellikkattil, Danielle Lemmon, Travis Allen O'Brien, June-Yi Lee, and Jung-Eun Chu
Geosci. Model Dev., 17, 301–320, https://doi.org/10.5194/gmd-17-301-2024, https://doi.org/10.5194/gmd-17-301-2024, 2024
Short summary
Short summary
This study introduces a new computational framework called Scalable Feature Extraction and Tracking (SCAFET), designed to extract and track features in climate data. SCAFET stands out by using innovative shape-based metrics to identify features without relying on preconceived assumptions about the climate model or mean state. This approach allows more accurate comparisons between different models and scenarios.
Min-Seop Ahn, Paul A. Ullrich, Peter J. Gleckler, Jiwoo Lee, Ana C. Ordonez, and Angeline G. Pendergrass
Geosci. Model Dev., 16, 3927–3951, https://doi.org/10.5194/gmd-16-3927-2023, https://doi.org/10.5194/gmd-16-3927-2023, 2023
Short summary
Short summary
We introduce a framework for regional-scale evaluation of simulated precipitation distributions with 62 climate reference regions and 10 metrics and apply it to evaluate CMIP5 and CMIP6 models against multiple satellite-based precipitation products. The common model biases identified in this study are mainly associated with the overestimated light precipitation and underestimated heavy precipitation. These biases persist from earlier-generation models and have been slightly improved in CMIP6.
Qi Tang, Jean-Christophe Golaz, Luke P. Van Roekel, Mark A. Taylor, Wuyin Lin, Benjamin R. Hillman, Paul A. Ullrich, Andrew M. Bradley, Oksana Guba, Jonathan D. Wolfe, Tian Zhou, Kai Zhang, Xue Zheng, Yunyan Zhang, Meng Zhang, Mingxuan Wu, Hailong Wang, Cheng Tao, Balwinder Singh, Alan M. Rhoades, Yi Qin, Hong-Yi Li, Yan Feng, Yuying Zhang, Chengzhu Zhang, Charles S. Zender, Shaocheng Xie, Erika L. Roesler, Andrew F. Roberts, Azamat Mametjanov, Mathew E. Maltrud, Noel D. Keen, Robert L. Jacob, Christiane Jablonowski, Owen K. Hughes, Ryan M. Forsyth, Alan V. Di Vittorio, Peter M. Caldwell, Gautam Bisht, Renata B. McCoy, L. Ruby Leung, and David C. Bader
Geosci. Model Dev., 16, 3953–3995, https://doi.org/10.5194/gmd-16-3953-2023, https://doi.org/10.5194/gmd-16-3953-2023, 2023
Short summary
Short summary
High-resolution simulations are superior to low-resolution ones in capturing regional climate changes and climate extremes. However, uniformly reducing the grid size of a global Earth system model is too computationally expensive. We provide an overview of the fully coupled regionally refined model (RRM) of E3SMv2 and document a first-of-its-kind set of climate production simulations using RRM at an economic cost. The key to this success is our innovative hybrid time step method.
Abhishekh Kumar Srivastava, Paul Aaron Ullrich, Deeksha Rastogi, Pouya Vahmani, Andrew Jones, and Richard Grotjahn
Geosci. Model Dev., 16, 3699–3722, https://doi.org/10.5194/gmd-16-3699-2023, https://doi.org/10.5194/gmd-16-3699-2023, 2023
Short summary
Short summary
Stakeholders need high-resolution regional climate data for applications such as assessing water availability and mountain snowpack. This study examines 3 h and 24 h historical precipitation over the contiguous United States in the 12 km WRF version 4.2.1-based dynamical downscaling of the ERA5 reanalysis. WRF improves precipitation characteristics such as the annual cycle and distribution of the precipitation maxima, but it also displays regionally and seasonally varying precipitation biases.
Zeyu Xue, Paul Ullrich, and Lai-Yung Ruby Leung
Hydrol. Earth Syst. Sci., 27, 1909–1927, https://doi.org/10.5194/hess-27-1909-2023, https://doi.org/10.5194/hess-27-1909-2023, 2023
Short summary
Short summary
We examine the sensitivity and robustness of conclusions drawn from the PGW method over the NEUS by conducting multiple PGW experiments and varying the perturbation spatial scales and choice of perturbed meteorological variables to provide a guideline for this increasingly popular regional modeling method. Overall, we recommend PGW experiments be performed with perturbations to temperature or the combination of temperature and wind at the gridpoint scale, depending on the research question.
David H. Marsico and Paul A. Ullrich
Geosci. Model Dev., 16, 1537–1551, https://doi.org/10.5194/gmd-16-1537-2023, https://doi.org/10.5194/gmd-16-1537-2023, 2023
Short summary
Short summary
Climate models involve several different components, such as the atmosphere, ocean, and land models. Information needs to be exchanged, or remapped, between these models, and devising algorithms for performing this exchange is important for ensuring the accuracy of climate simulations. In this paper, we examine the efficacy of several traditional and novel approaches to remapping on the sphere and demonstrate where our approaches offer improvement.
Chengzhu Zhang, Jean-Christophe Golaz, Ryan Forsyth, Tom Vo, Shaocheng Xie, Zeshawn Shaheen, Gerald L. Potter, Xylar S. Asay-Davis, Charles S. Zender, Wuyin Lin, Chih-Chieh Chen, Chris R. Terai, Salil Mahajan, Tian Zhou, Karthik Balaguru, Qi Tang, Cheng Tao, Yuying Zhang, Todd Emmenegger, Susannah Burrows, and Paul A. Ullrich
Geosci. Model Dev., 15, 9031–9056, https://doi.org/10.5194/gmd-15-9031-2022, https://doi.org/10.5194/gmd-15-9031-2022, 2022
Short summary
Short summary
Earth system model (ESM) developers run automated analysis tools on data from candidate models to inform model development. This paper introduces a new Python package, E3SM Diags, that has been developed to support ESM development and use routinely in the development of DOE's Energy Exascale Earth System Model. This tool covers a set of essential diagnostics to evaluate the mean physical climate from simulations, as well as several process-oriented and phenomenon-based evaluation diagnostics.
Vijay S. Mahadevan, Jorge E. Guerra, Xiangmin Jiao, Paul Kuberry, Yipeng Li, Paul Ullrich, David Marsico, Robert Jacob, Pavel Bochev, and Philip Jones
Geosci. Model Dev., 15, 6601–6635, https://doi.org/10.5194/gmd-15-6601-2022, https://doi.org/10.5194/gmd-15-6601-2022, 2022
Short summary
Short summary
Coupled Earth system models require transfer of field data between multiple components with varying spatial resolutions to determine the correct climate behavior. We present the Metrics for Intercomparison of Remapping Algorithms (MIRA) protocol to evaluate the accuracy, conservation properties, monotonicity, and local feature preservation of four different remapper algorithms for various unstructured mesh problems of interest. Future extensions to more practical use cases are also discussed.
Paul A. Ullrich, Colin M. Zarzycki, Elizabeth E. McClenny, Marielle C. Pinheiro, Alyssa M. Stansfield, and Kevin A. Reed
Geosci. Model Dev., 14, 5023–5048, https://doi.org/10.5194/gmd-14-5023-2021, https://doi.org/10.5194/gmd-14-5023-2021, 2021
Short summary
Short summary
TempestExtremes (TE) is a multifaceted framework for feature detection, tracking, and scientific analysis of regional or global Earth system datasets. Version 2.1 of TE now provides extensive support for nodal and areal features. This paper describes the algorithms that have been added to the TE framework since version 1.0 and gives several examples of how these can be combined to produce composite algorithms for evaluating and understanding atmospheric features.
Prabhat, Karthik Kashinath, Mayur Mudigonda, Sol Kim, Lukas Kapp-Schwoerer, Andre Graubner, Ege Karaismailoglu, Leo von Kleist, Thorsten Kurth, Annette Greiner, Ankur Mahesh, Kevin Yang, Colby Lewis, Jiayi Chen, Andrew Lou, Sathyavat Chandran, Ben Toms, Will Chapman, Katherine Dagon, Christine A. Shields, Travis O'Brien, Michael Wehner, and William Collins
Geosci. Model Dev., 14, 107–124, https://doi.org/10.5194/gmd-14-107-2021, https://doi.org/10.5194/gmd-14-107-2021, 2021
Short summary
Short summary
Detecting extreme weather events is a crucial step in understanding how they change due to climate change. Deep learning (DL) is remarkable at pattern recognition; however, it works best only when labeled datasets are available. We create
ClimateNet– an expert-labeled curated dataset – to train a DL model for detecting weather events and predicting changes in extreme precipitation. This work paves the way for DL-based automated, high-fidelity, and highly precise analytics of climate data.
Travis A. O'Brien, Mark D. Risser, Burlen Loring, Abdelrahman A. Elbashandy, Harinarayan Krishnan, Jeffrey Johnson, Christina M. Patricola, John P. O'Brien, Ankur Mahesh, Prabhat, Sarahí Arriaga Ramirez, Alan M. Rhoades, Alexander Charn, Héctor Inda Díaz, and William D. Collins
Geosci. Model Dev., 13, 6131–6148, https://doi.org/10.5194/gmd-13-6131-2020, https://doi.org/10.5194/gmd-13-6131-2020, 2020
Short summary
Short summary
Researchers utilize various algorithms to identify extreme weather features in climate data, and we seek to answer this question: given a
plausibleweather event detector, how does uncertainty in the detector impact scientific results? We generate a suite of statistical models that emulate expert identification of weather features. We find that the connection between El Niño and atmospheric rivers – a specific extreme weather type – depends systematically on the design of the detector.
Cited articles
Algarra, I., Nieto, R., Ramos, A. M., Eiras-Barca, J., Trigo, R. M., and Gimeno, L.: Significant increase of global anomalous moisture uptake feeding land-falling atmospheric rivers, Nat. Commun., 11, 5082, https://doi.org/10.1038/s41467-020-18876-w, 2020.
Beck, H. E., van Dijk, A. I. J. M., Levizzani, V., Schellekens, J., Miralles, D. G., Martens, B., and de Roo, A.: MSWEP: 3-hourly 0.25° global gridded precipitation (1979–2015) by merging gauge, satellite, and reanalysis data, Hydrol. Earth Syst. Sci., 21, 589–615, https://doi.org/10.5194/hess-21-589-2017, 2017.
Bracegirdle, T. J., Holmes, C. R., Hosking, J. S., Marshall, G. J., Osman, M., Patterson, M., and Rackow, T.: Improvements in circumpolar Southern Hemisphere extratropical atmospheric circulation in CMIP6 compared to CMIP5, Earth Space Sci. 7, e2019EA001065, https://doi.org/10.1029/2019EA001065, 2020.
Buizza, R., Poli, P., Rixen, M., Alonso-Balmaseda, M., Bosilovich, M. G., Brönnimann, S., and Vaselali, A.: Advancing global and regional reanalyses, B. Am. Meteorol. Soc., 99, ES139–ES144, 2018.
Caldwell, P. M., Mametjanov, A., Tang, Q., Van Roekel, L. P., Golaz, J. C., Lin, W., and Zhou, T.: The DOE E3SM coupled model version 1: Description and results at high resolution, J. Adv. Model. Earth Sy., 11, 4095–4146, 2019.
Chapman, W. E., Subramanian, A. C., Delle Monache, L., Xie, S. P., and Ralph, F. M.: Improving atmospheric river forecasts with machine learning, Geophys. Res. Lett., 46, 10627–10635, 2019.
DeFlorio, M. J., Waliser, D. E., Guan, B., Lavers, D. A., Ralph, F. M., and Vitart, F.: Global assessment of atmospheric river prediction skill, J. Hydrometeorol., 19, 409–426, 2018.
Dettinger, M. D., Ralph, F. M., Das, T., Neiman, P. J., and Cayan, D. R.: Atmospheric rivers, floods and the water resources of California, Water, 3, 445–478, 2011.
Dong, B., Lee, J. Ullrich, P., Gleckler, P., and Kristin, C.: PCMDI/ARMP: v0.1.0 (v0.1.0), Zenodo [code], https://doi.org/10.5281/zenodo.14188790, 2024.
Eyring, V., Bony, S., Meehl, G. A., Senior, C. A., Stevens, B., Stouffer, R. J., and Taylor, K. E.: Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization, Geosci. Model Dev., 9, 1937–1958, https://doi.org/10.5194/gmd-9-1937-2016, 2016.
Gelaro, R., McCarty, W., Suárez, M. J., Todling, R., Molod, A., Takacs, L., Randles, C. A., Darmenov, A., Bosilovich, M. G., Reichle, R., and Wargan, K.: The modern-era retrospective analysis for research and applications, version 2 (MERRA-2), J. Climate, 30, 5419–5454, 2017.
Gershunov, A., Shulgina, T., Clemesha, R. E., Guirguis, K., Pierce, D. W., Dettinger, M. D., Lavers, D. A., Cayan, D. R., Polade, S. D., Kalansky, J., and Ralph, F. M.: Precipitation regime change in western North America: the role of atmospheric rivers, Sci. Rep., 9, 9944, https://doi.org/10.1029/2020JD032701, 2019.
Gimeno, L., Nieto, R., Vázquez, M., and Lavers, D. A.: Atmospheric rivers: A mini-review, Front. Earth Sci., 2, https://doi.org/10.3389/feart.2014.00002, 2014.
Gleckler, P. J., Taylor, K. E., and Doutriaux, C.: Performance metrics for climate models, J. Geophys. Res.-Atmos., 113, D06104, https://doi.org/10.1029/2007JD008972, 2008.
Golaz, J. C., Caldwell, P. M., Van Roekel, L. P., Petersen, M. R., Tang, Q., Wolfe, J. D., and Zhu, Q.: The DOE E3SM coupled model version 1: Overview and evaluation at standard resolution, J. Adv. Model. Earth Sy., 11, 2089–2129, 2019.
Griffith, D. A.: Establishing qualitative geographic sample size in the presence of spatial autocorrelation, Ann. Assoc. Am. Geogr., 103, 1107–1122, 2013.
Guan, B. and Waliser, D. E.: Atmospheric rivers in 20-year weather and climate simulations: A multimodel, global evaluation, J. Geophys. Res.-Atmos., 122, 5556–5581, 2017.
Guan, B., Waliser, D. E., and Ralph, F. M.: Global application of the atmospheric river scale, J. Geophys. Res.-Atmos., 128, e2022JD037180, https://doi.org/10.1029/2022JD037180, 2023.
Harrop, B., Leung, L., and Ullrich, P.: Improving Simulations of Atmospheric Rivers and Heat Waves in the Coupled E3SM, FY2023 First Quarter Performance Metric, DOE/SC-CM-23-001, 2023.
Harvey, B. J., Cook, P., Shaffrey, L. C., and Schiemann, R.: The response of the northern hemisphere storm tracks and jet streams to climate change in the CMIP3, CMIP5, and CMIP6 climate models, J. Geophys. Res.-Atmos., 125, e2020JD032701, https://doi.org/10.1126/sciadv.aba132, 2020.
Hersbach, H., Bell, B., Berrisford, P., Hirahara, S., Horányi, A., Muñoz-Sabater, J., Nicolas, J., Peubey, C., Radu, R., Schepers, D., and Simmons, A.: The ERA5 global reanalysis, Q. J. Roy. Meteor. Soc., 146, 1999–2049, 2020.
Hoyer, S. and Hamman, J.: xarray: N-D labeled Arrays and Datasets in Python, J. Open Res. Softw., 5, 10, https://doi.org/10.5334/jors.148, 2017.
Huang, X., Swain, D. L., and Hall, A. D.: Future precipitation increase from very high resolution ensemble downscaling of extreme atmospheric river storms in California, Sci. Adv., 6, eaba1323, https://doi.org/10.1038/s41598-019-46169-w, 2020.
Inda-Díaz, H. A., O'Brien, T. A., Zhou, Y., and Collins, W. D.: Constraining and characterizing the size of atmospheric rivers: A perspective independent from the detection algorithm, J. Geophys. Res.-Atmos., 126, e2020JD033746, https://doi.org/10.1029/2020JD033746, 2021.
Jenkins, G. S., Adamou, G., and Fongang, S.: The challenges of modeling climate variability and change in West Africa, Climatic Change, 52, 263–286, 2002.
Kobayashi, S., Ota, Y., Harada, Y., Ebita, A., Moriya, M., Onoda, H., Kamahori, H., Kobayashi, C., Endo, H., and Miyaoka, K.: The JRA-55 reanalysis: General specifications and basic characteristics, J. Meteorol. Soc. Jpn., 93, 5–48, https://doi.org/10.2151/jmsj.2015-001, 2015.
Lee, J., Gleckler, P. J., Ahn, M.-S., Ordonez, A., Ullrich, P. A., Sperber, K. R., Taylor, K. E., Planton, Y. Y., Guilyardi, E., Durack, P., Bonfils, C., Zelinka, M. D., Chao, L.-W., Dong, B., Doutriaux, C., Zhang, C., Vo, T., Boutte, J., Wehner, M. F., Pendergrass, A. G., Kim, D., Xue, Z., Wittenberg, A. T., and Krasting, J.: Systematic and objective evaluation of Earth system models: PCMDI Metrics Package (PMP) version 3, Geosci. Model Dev., 17, 3919–3948, https://doi.org/10.5194/gmd-17-3919-2024, 2024.
Leung, L. R. and Qian, Y.: Atmospheric rivers induced heavy precipitation and flooding in the western US simulated by the WRF regional climate model, Geophys. Res. Lett., 36, L03820, https://doi.org/10.1029/2008GL036445, 2009.
Massoud, E. C., Espinoza, V., Guan, B., and Waliser, D. E.: Global Climate Model Ensemble Approaches for Future Projections of Atmospheric Rivers, Earth's Future, 7, 1136–1151, 2019.
Monerie, P.-A., Wainwright, C. M., Sidibe, M., and Afolayan Akinsanola, A.: Model uncertainties in climate change impacts on Sahel precipitation in ensembles of CMIP5 and CMIP6 simulations, Clim. Dynam., 55, 1385–1401, 2020.
Mundhenk, B. D., Barnes, E. A., and Maloney, E. D.: All-season climatology and variability of atmospheric river frequencies over the North Pacific, J. Climate, 29, 4885–4903, 2016.
Nardi, K. M., Barnes, E. A. and Ralph, F. M.: Assessment of numerical weather prediction model reforecasts of the occurrence, intensity, and location of atmospheric rivers along the West Coast of North America, Mon. Weather Rev., 146, 3343–3362, 2018.
Neiman, P. J., White, A. B., Ralph, F. M., Gottas, D. J., and Gutman, S. I.: A water vapour flux tool for precipitation forecasting, Proc. Inst. Civil Eng. Water Manag., 162, 83–94, 2009.
Neiman, P. J., Schick, L. J., Ralph, F. M., Hughes, M., and Wick, G. A.: Flooding in western Washington: The connection to atmospheric rivers, J. Hydrometeorol., 12, 1337–1358, 2011.
Neiman, P. J., Ralph, F. M., Moore, B. J., Hughes, M., Mahoney, K. M., Cordeira, J. M., and Dettinger, M. D.: The landfall and inland penetration of a flood-producing atmospheric river in Arizona. Part I: Observed synoptic-scale, orographic, and hydrometeorological characteristics, J. Hydrometeorol., 14, 460–484, 2013.
Newman, M., Kiladis, G. N., Weickmann, K. M., Ralph, F. M., and Sardeshmukh, P. D.: Relative contributions of synoptic and low-frequency eddies to time-mean atmospheric moisture transport, including the role of atmospheric rivers, J. Climate, 25, 7341–7361, 2012.
O'Brien, T., Wehner, M., Payne, A., Shields, C., Rutz, J., Leung, L.-R., Ralph, F. M., Collow, A., Gorodetskaya, I., Guan, B., and Lora, J. M.: Increases in Future AR Count and Size: Overview of the ARTMIP Tier 2 CMIP5/6 Experiment, J. Geophys. Res.-Atmos., 127, e2021JD036013, https://doi.org/10.1029/2021JD036013, 2022.
Payne, A. E., Demory, M. E., Leung, L. R., Ramos, A. M., Shields, C. A., Rutz, J. J., Siler, N., Villarini, G., Hall, A., and Ralph, F. M.: Responses and impacts of atmospheric rivers to climate change, Nat. Rev. Earth Environ., 1, 143–157, https://doi.org/10.1038/s43017-020-0030-5, 2020.
Ralph, F., Coleman, T., Neiman, P., Zamora, R., and Dettinger, M.: Observed impacts of duration and seasonality of atmospheric-river landfalls on soil moisture and runoff in coastal northern California, J. Hydrometeorol., 14, 443–459, 2013.
Ralph, F. M. and Dettinger, M.: Storms, floods, and the science of atmospheric rivers, Eos T. Am. Geophys. Union, 92, 265–266, 2011.
Ralph, F. M., Neiman, P. J., Wick, G. A., Gutman, S. I., Dettinger, M. D., Cayan, D. R., and White, A. B.: Flooding on California's Russian River: Role of atmospheric rivers, Geophys. Res. Lett., 33, L13801, https://doi.org/10.1029/2006GL026689, 2006.
Ralph, F. M., Dettinger, M. D., Cairns, M. M., Galarneau, T. J., and Eylander, J.: Defining “atmospheric river”: How the Glossary of Meteorology helped resolve a debate, B. Am. Meteorol. Soc., 99, 837–839, 2018.
Rutz, J. J., Shields, C. A., Lora, J. M., Payne, A. E., Guan, B., Ullrich, P., O'Brien, T., Leung, L. R., Ralph, F. M., Wehner, M., and Brands, S.: The atmospheric river tracking method intercomparison project (ARTMIP): Quantifying uncertainties in atmospheric river climatology, J. Geophys. Res.-Atmos., 124, 13777–13802, 2019.
Shields, C. A., Rosenbloom, N., Bates, S., Hannay, C., Hu, A., Payne, A. E. Rutz, J. J., and Truesdale, J.: Meridional heat transport during atmospheric rivers in high-resolution CESM climate projections, Geophys. Res. Lett., 46, 14702–14712, 2019a.
Shields, C. A., Rutz, J. J., Leung, L. R., Ralph, F. M., Wehner, M., O'Brien, T., and Pierce, R.: Defining uncertainties through comparison of atmospheric river tracking methods, B. Am. Meteorol. Soc., 100, ES93–ES96, 2019b.
Skinner, C. B., Lora, J. M., Payne, A. E., and Poulsen, C. J.: Atmospheric river changes shaped mid-latitude hydroclimate since the mid-Holocene, Earth Planet. Sc. Lett., 541, 116293, https://doi.org/10.1016/j.epsl.2020.116293, 2020.
Skinner, C. B., Lora, J. M., Tabor, C., and Zhu, J.: Atmospheric river contributions to ice sheet hydroclimate at the Last Glacial Maximum. Geophys. Res. Lett., 50, e2022GL101750, https://doi.org/10.1029/2022gl101750, 2023.
Taylor, K. E.: Summarizing multiple aspects of model performance in a single diagram, J. Geophys. Res.-Atmos., 106, 7183–7192, 2001.
Ullrich, P. A. and Zarzycki, C. M.: TempestExtremes: a framework for scale-insensitive pointwise feature tracking on unstructured grids, Geosci. Model Dev., 10, 1069–1090, https://doi.org/10.5194/gmd-10-1069-2017, 2017.
Ullrich, P. A., Zarzycki, C. M., McClenny, E. E., Pinheiro, M. C., Stansfield, A. M., and Reed, K. A.: TempestExtremes v2.1: a community framework for feature detection, tracking, and analysis in large datasets, Geosci. Model Dev., 14, 5023–5048, https://doi.org/10.5194/gmd-14-5023-2021, 2021.
Vo, J., Gagne, D., Smith, B., Bart, M., and Maloney, E. D.: xCDAT: A Python Package for Simple and Robust Analysis of Climate Data, J. Open Source Softw., 9, 6426, https://doi.org/10.21105/joss.06426, 2024.
Whittleston, D., Nicholson, S. E., Schlosser, A., and Entekhabi, D.: Climate models lack jet–rainfall coupling over West Africa, J. Climate, 30, 4625–4632, 2017.
Wick, G. A., Paul, J. N., Ralph, F. M., and Hamill, T. M.: Evaluation of forecasts of the water vapor signature of atmospheric rivers in operational numerical weather prediction models, Weather Forecast., 28, 1337–1352, 2013.
Wilks, D. S.: Statistical methods in the atmospheric sciences, Vol. 100, Academic Press, ISBN 978-0-12-815823-4, 2011.
Zarzycki, C. M., Ullrich, P. A., and Reed, K. A.: Metrics for evaluating tropical cyclones in climate data, J. Appl. Meteorol. Clim., 60, 643–660, 2021.
Zhang, L., Zhao, Y., Cheng, T. F., and Lu, M.: Future changes in global atmospheric rivers projected by CMIP6 models, J. Geophys. Res.-Atmos., 129, e2023JD039359, https://doi.org/10.1029/2023JD039359, 2024.
Zhu, Y. and Newell, R. E.: A proposed algorithm for moisture fluxes from atmospheric rivers, Mon. Weather Rev., 126, 725–735, 1998.
Short summary
A metrics package designed for easy analysis of atmospheric river (AR) characteristics and statistics is presented. The tool is efficient for diagnosing systematic AR bias in climate models and useful for evaluating new AR characteristics in model simulations. In climate models, landfalling AR precipitation shows dry biases globally, and AR tracks are farther poleward (equatorward) in the North and South Atlantic (South Pacific and Indian Ocean).
A metrics package designed for easy analysis of atmospheric river (AR) characteristics and...
