Articles | Volume 17, issue 9
https://doi.org/10.5194/gmd-17-3919-2024
© Author(s) 2024. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/gmd-17-3919-2024
© Author(s) 2024. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Model evaluation paper
| 
15 May 2024
Model evaluation paper |
| 15 May 2024
| 15 May 2024
Systematic and objective evaluation of Earth system models: PCMDI Metrics Package (PMP) version 3
Lawrence Livermore National Laboratory, Livermore, CA, USA
Peter J. Gleckler
Lawrence Livermore National Laboratory, Livermore, CA, USA
Min-Seop Ahn
NASA Goddard Space Flight Center, Greenbelt, MD, USA
ESSIC, University of Maryland, College Park, MD, USA
Ana Ordonez
Lawrence Livermore National Laboratory, Livermore, CA, USA
Paul A. Ullrich
Lawrence Livermore National Laboratory, Livermore, CA, USA
Department of Land, Air and Water Resources, University of California, Davis, Davis, CA, USA
Kenneth R. Sperber
Lawrence Livermore National Laboratory, Livermore, CA, USA
retired
Karl E. Taylor
Lawrence Livermore National Laboratory, Livermore, CA, USA
Yann Y. Planton
NOAA Pacific Marine Environmental Laboratory, Seattle, WA, USA
School of Earth Atmosphere and Environment, Monash University, Clayton, VIC, Australia
Eric Guilyardi
LOCEAN-IPSL, CNRS-IRD-MNHN-Sorbonne Université, Paris, France
National Centre for Atmospheric Science – Climate, University of Reading, Reading, UK
Paul Durack
Lawrence Livermore National Laboratory, Livermore, CA, USA
Celine Bonfils
Lawrence Livermore National Laboratory, Livermore, CA, USA
Mark D. Zelinka
Lawrence Livermore National Laboratory, Livermore, CA, USA
Li-Wei Chao
Lawrence Livermore National Laboratory, Livermore, CA, USA
Bo Dong
Lawrence Livermore National Laboratory, Livermore, CA, USA
Charles Doutriaux
Lawrence Livermore National Laboratory, Livermore, CA, USA
Chengzhu Zhang
Lawrence Livermore National Laboratory, Livermore, CA, USA
Tom Vo
Lawrence Livermore National Laboratory, Livermore, CA, USA
Jason Boutte
Lawrence Livermore National Laboratory, Livermore, CA, USA
Michael F. Wehner
Lawrence Berkeley National Laboratory, Berkeley, CA, USA
Angeline G. Pendergrass
Department of Earth and Atmospheric Science, Cornell University, Ithaca, NY, USA
National Center for Atmospheric Research, Boulder, CO, USA
Daehyun Kim
School of Earth and Environmental Sciences, Seoul National University, Seoul, South Korea
Pacific Northwest National Laboratory, Richland, WA, USA
Andrew T. Wittenberg
NOAA Geophysical Fluid Dynamics Laboratory, Princeton, NJ, USA
John Krasting
NOAA Geophysical Fluid Dynamics Laboratory, Princeton, NJ, USA
Related authors
Forrest M. Hoffman, Birgit Hassler, Ranjini Swaminathan, Jared Lewis, Bouwe Andela, Nathaniel Collier, Dóra Hegedűs, Jiwoo Lee, Charlotte Pascoe, Mika Pflüger, Martina Stockhause, Paul Ullrich, Min Xu, Lisa Bock, Felicity Chun, Bettina K. Gier, Douglas I. Kelley, Axel Lauer, Julien Lenhardt, Manuel Schlund, Mohanan G. Sreeush, Katja Weigel, Ed Blockley, Rebecca Beadling, Romain Beucher, Demiso D. Dugassa, Valerio Lembo, Jianhua Lu, Swen Brands, Jerry Tjiputra, Elizaveta Malinina, Brian Mederios, Enrico Scoccimarro, Jeremy Walton, Philip Kershaw, André L. Marquez, Malcolm J. Roberts, Eleanor O’Rourke, Elisabeth Dingley, Briony Turner, Helene Hewitt, and John P. Dunne
EGUsphere, https://doi.org/10.5194/egusphere-2025-2685, https://doi.org/10.5194/egusphere-2025-2685, 2025
This preprint is open for discussion and under review for Geoscientific Model Development (GMD).
Short summary
Short summary
As Earth system models become more complex, rapid and comprehensive evaluation through comparison with observational data is necessary. The upcoming Assessment Fast Track for the Seventh Phase of the Coupled Model Intercomparison Project (CMIP7) will require fast analysis. This paper describes a new Rapid Evaluation Framework (REF) that was developed for the Assessment Fast Track that will be run at the Earth System Grid Federation (ESGF) to inform the community about the performance of models.
Katherine M. Smith, Alice M. Barthel, LeAnn M. Conlon, Luke P. Van Roekel, Anthony Bartoletti, Jean-Christophe Golaz, Chengzhu Zhang, Carolyn Branecky Begeman, James J. Benedict, Gautam Bisht, Yan Feng, Walter Hannah, Bryce E. Harrop, Nicole Jeffery, Wuyin Lin, Po-Lun Ma, Mathew E. Maltrud, Mark R. Petersen, Balwinder Singh, Qi Tang, Teklu Tesfa, Jonathan D. Wolfe, Shaocheng Xie, Xue Zheng, Karthik Balaguru, Oluwayemi Garuba, Peter Gleckler, Aixue Hu, Jiwoo Lee, Ben Moore-Maley, and Ana C. Ordoñez
Geosci. Model Dev., 18, 1613–1633, https://doi.org/10.5194/gmd-18-1613-2025, https://doi.org/10.5194/gmd-18-1613-2025, 2025
Short summary
Short summary
Version 2.1 of the U.S. Department of Energy's Energy Exascale Earth System Model (E3SM) adds the Fox-Kemper et al. (2011) mixed-layer eddy parameterization, which restratifies the ocean surface layer through an overturning streamfunction. Results include surface layer bias reduction in temperature, salinity, and sea ice extent in the North Atlantic; a small strengthening of the Atlantic meridional overturning circulation; and improvements to many atmospheric climatological variables.
Bo Dong, Paul Ullrich, Jiwoo Lee, Peter Gleckler, Kristin Chang, and Travis A. O'Brien
Geosci. Model Dev., 18, 961–976, https://doi.org/10.5194/gmd-18-961-2025, https://doi.org/10.5194/gmd-18-961-2025, 2025
Short summary
Short summary
A metrics package designed for easy analysis of atmospheric river (AR) characteristics and statistics is presented. The tool is efficient for diagnosing systematic AR bias in climate models and useful for evaluating new AR characteristics in model simulations. In climate models, landfalling AR precipitation shows dry biases globally, and AR tracks are farther poleward (equatorward) in the North and South Atlantic (South Pacific and Indian Ocean).
Paul J. Durack, Karl E. Taylor, Peter J. Gleckler, Gerald A. Meehl, Bryan N. Lawrence, Curt Covey, Ronald J. Stouffer, Guillaume Levavasseur, Atef Ben-Nasser, Sebastien Denvil, Martina Stockhause, Jonathan M. Gregory, Martin Juckes, Sasha K. Ames, Fabrizio Antonio, David C. Bader, John P. Dunne, Daniel Ellis, Veronika Eyring, Sandro L. Fiore, Sylvie Joussaume, Philip Kershaw, Jean-Francois Lamarque, Michael Lautenschlager, Jiwoo Lee, Chris F. Mauzey, Matthew Mizielinski, Paola Nassisi, Alessandra Nuzzo, Eleanor O’Rourke, Jeffrey Painter, Gerald L. Potter, Sven Rodriguez, and Dean N. Williams
EGUsphere, https://doi.org/10.5194/egusphere-2024-3729, https://doi.org/10.5194/egusphere-2024-3729, 2025
Short summary
Short summary
CMIP6 was the most expansive and ambitious Model Intercomparison Project (MIP), the latest in a history, extending four decades. CMIP engaged a growing community focused on improving climate understanding, and quantifying and attributing observed climate change being experienced today. The project's profound impact is due to the combining the latest climate science and technology, enabling the latest-generation climate simulations and increasing community attention in every successive phase.
Seung H. Baek, Paul A. Ullrich, Bo Dong, and Jiwoo Lee
Geosci. Model Dev., 17, 8665–8681, https://doi.org/10.5194/gmd-17-8665-2024, https://doi.org/10.5194/gmd-17-8665-2024, 2024
Short summary
Short summary
We evaluate downscaled products by examining locally relevant co-variances during precipitation events. Common statistical downscaling techniques preserve expected co-variances during convective precipitation (a stationary phenomenon). However, they dampen future intensification of frontal precipitation (a non-stationary phenomenon) captured in global climate models and dynamical downscaling. Our study quantifies a ramification of the stationarity assumption underlying statistical downscaling.
Min-Seop Ahn, Paul A. Ullrich, Peter J. Gleckler, Jiwoo Lee, Ana C. Ordonez, and Angeline G. Pendergrass
Geosci. Model Dev., 16, 3927–3951, https://doi.org/10.5194/gmd-16-3927-2023, https://doi.org/10.5194/gmd-16-3927-2023, 2023
Short summary
Short summary
We introduce a framework for regional-scale evaluation of simulated precipitation distributions with 62 climate reference regions and 10 metrics and apply it to evaluate CMIP5 and CMIP6 models against multiple satellite-based precipitation products. The common model biases identified in this study are mainly associated with the overestimated light precipitation and underestimated heavy precipitation. These biases persist from earlier-generation models and have been slightly improved in CMIP6.
Forrest M. Hoffman, Birgit Hassler, Ranjini Swaminathan, Jared Lewis, Bouwe Andela, Nathaniel Collier, Dóra Hegedűs, Jiwoo Lee, Charlotte Pascoe, Mika Pflüger, Martina Stockhause, Paul Ullrich, Min Xu, Lisa Bock, Felicity Chun, Bettina K. Gier, Douglas I. Kelley, Axel Lauer, Julien Lenhardt, Manuel Schlund, Mohanan G. Sreeush, Katja Weigel, Ed Blockley, Rebecca Beadling, Romain Beucher, Demiso D. Dugassa, Valerio Lembo, Jianhua Lu, Swen Brands, Jerry Tjiputra, Elizaveta Malinina, Brian Mederios, Enrico Scoccimarro, Jeremy Walton, Philip Kershaw, André L. Marquez, Malcolm J. Roberts, Eleanor O’Rourke, Elisabeth Dingley, Briony Turner, Helene Hewitt, and John P. Dunne
EGUsphere, https://doi.org/10.5194/egusphere-2025-2685, https://doi.org/10.5194/egusphere-2025-2685, 2025
This preprint is open for discussion and under review for Geoscientific Model Development (GMD).
Short summary
Short summary
As Earth system models become more complex, rapid and comprehensive evaluation through comparison with observational data is necessary. The upcoming Assessment Fast Track for the Seventh Phase of the Coupled Model Intercomparison Project (CMIP7) will require fast analysis. This paper describes a new Rapid Evaluation Framework (REF) that was developed for the Assessment Fast Track that will be run at the Earth System Grid Federation (ESGF) to inform the community about the performance of models.
Rachel Yuen Sum Tam, Timothy Myers, Mark Zelinka, Cristian Proistosescu, Yuan-Jen Lin, and Kate Marvel
EGUsphere, https://doi.org/10.5194/egusphere-2025-3177, https://doi.org/10.5194/egusphere-2025-3177, 2025
Short summary
Short summary
This work identifies the key driver to the change of present and future climate response, known as the pattern effect, by breaking down low-cloud feedback as the radiative changes to meteorology and the meteorology changes to warming using a cloud controlling factor framework. We identify inversion strength in the Southern Ocean and the South East Pacific as the main driver to the pattern effect, and larger uncertainty remains in the sensitivities of radiative flux to meteorology.
Beth Dingley, James A. Anstey, Marta Abalos, Carsten Abraham, Tommi Bergman, Lisa Bock, Sonya Fiddes, Birgit Hassler, Ryan J. Kramer, Fei Luo, Fiona M. O'Connor, Petr Šácha, Isla R. Simpson, Laura J. Wilcox, and Mark D. Zelinka
EGUsphere, https://doi.org/10.5194/egusphere-2025-3189, https://doi.org/10.5194/egusphere-2025-3189, 2025
This preprint is open for discussion and under review for Geoscientific Model Development (GMD).
Short summary
Short summary
This manuscript defines as a list of variables and scientific opportunities which are requested from the CMIP7 Assessment Fast Track to address open atmospheric science questions. The list reflects the output of a large public community engagement effort, coordinated across autumn 2025 through to summer 2025.
Baylor Fox-Kemper, Patricia DeRepentigny, Anne Marie Treguier, Christian Stepanek, Eleanor O’Rourke, Chloe Mackallah, Alberto Meucci, Yevgeny Aksenov, Paul J. Durack, Nicole Feldl, Vanessa Hernaman, Céline Heuzé, Doroteaciro Iovino, Gaurav Madan, André L. Marquez, François Massonnet, Jenny Mecking, Dhrubajyoti Samanta, Patrick C. Taylor, Wan-Ling Tseng, and Martin Vancoppenolle
EGUsphere, https://doi.org/10.5194/egusphere-2025-3083, https://doi.org/10.5194/egusphere-2025-3083, 2025
This preprint is open for discussion and under review for Geoscientific Model Development (GMD).
Short summary
Short summary
The earth system model variables needed for studies of the ocean and sea ice are prioritized and requested.
Anna Zehrung, Andrew D. King, Zebedee Nicholls, Mark D. Zelinka, and Malte Meinshausen
EGUsphere, https://doi.org/10.5194/egusphere-2025-2252, https://doi.org/10.5194/egusphere-2025-2252, 2025
This preprint is open for discussion and under review for Geoscientific Model Development (GMD).
Short summary
Short summary
The Gregory method is a common approach for calculating the equilibrium climate sensitivity (ECS). However, studies which apply this method lack transparency in how model data is processed prior to calculating the ECS, inhibiting replicability. Different choices of global and annual mean weighting, anomaly calculation, and linear regression fit can affect the ECS estimates. We investigate the impact of these choices and propose a standardised method for future ECS calculations.
Martin Juckes, Karl E. Taylor, Fabrizio Antonio, David Brayshaw, Carlo Buontempo, Jian Cao, Paul J. Durack, Michio Kawamiya, Hyungjun Kim, Tomas Lovato, Chloe Mackallah, Matthew Mizielinski, Alessandra Nuzzo, Martina Stockhause, Daniele Visioni, Jeremy Walton, Briony Turner, Eleanor O'Rourke, and Beth Dingley
Geosci. Model Dev., 18, 2639–2663, https://doi.org/10.5194/gmd-18-2639-2025, https://doi.org/10.5194/gmd-18-2639-2025, 2025
Short summary
Short summary
The Baseline Climate Variables for Earth System Modelling (ESM-BCVs) are defined as a list of 135 variables which have high utility for the evaluation and exploitation of climate simulations. The list reflects the most frequently used variables from Earth system models based on an assessment of data publication and download records from the largest archive of global climate projects.
Katherine M. Smith, Alice M. Barthel, LeAnn M. Conlon, Luke P. Van Roekel, Anthony Bartoletti, Jean-Christophe Golaz, Chengzhu Zhang, Carolyn Branecky Begeman, James J. Benedict, Gautam Bisht, Yan Feng, Walter Hannah, Bryce E. Harrop, Nicole Jeffery, Wuyin Lin, Po-Lun Ma, Mathew E. Maltrud, Mark R. Petersen, Balwinder Singh, Qi Tang, Teklu Tesfa, Jonathan D. Wolfe, Shaocheng Xie, Xue Zheng, Karthik Balaguru, Oluwayemi Garuba, Peter Gleckler, Aixue Hu, Jiwoo Lee, Ben Moore-Maley, and Ana C. Ordoñez
Geosci. Model Dev., 18, 1613–1633, https://doi.org/10.5194/gmd-18-1613-2025, https://doi.org/10.5194/gmd-18-1613-2025, 2025
Short summary
Short summary
Version 2.1 of the U.S. Department of Energy's Energy Exascale Earth System Model (E3SM) adds the Fox-Kemper et al. (2011) mixed-layer eddy parameterization, which restratifies the ocean surface layer through an overturning streamfunction. Results include surface layer bias reduction in temperature, salinity, and sea ice extent in the North Atlantic; a small strengthening of the Atlantic meridional overturning circulation; and improvements to many atmospheric climatological variables.
Malcolm J. Roberts, Kevin A. Reed, Qing Bao, Joseph J. Barsugli, Suzana J. Camargo, Louis-Philippe Caron, Ping Chang, Cheng-Ta Chen, Hannah M. Christensen, Gokhan Danabasoglu, Ivy Frenger, Neven S. Fučkar, Shabeh ul Hasson, Helene T. Hewitt, Huanping Huang, Daehyun Kim, Chihiro Kodama, Michael Lai, Lai-Yung Ruby Leung, Ryo Mizuta, Paulo Nobre, Pablo Ortega, Dominique Paquin, Christopher D. Roberts, Enrico Scoccimarro, Jon Seddon, Anne Marie Treguier, Chia-Ying Tu, Paul A. Ullrich, Pier Luigi Vidale, Michael F. Wehner, Colin M. Zarzycki, Bosong Zhang, Wei Zhang, and Ming Zhao
Geosci. Model Dev., 18, 1307–1332, https://doi.org/10.5194/gmd-18-1307-2025, https://doi.org/10.5194/gmd-18-1307-2025, 2025
Short summary
Short summary
HighResMIP2 is a model intercomparison project focusing on high-resolution global climate models, that is, those with grid spacings of 25 km or less in the atmosphere and ocean, using simulations of decades to a century in length. We are proposing an update of our simulation protocol to make the models more applicable to key questions for climate variability and hazard in present-day and future projections and to build links with other communities to provide more robust climate information.
Bo Dong, Paul Ullrich, Jiwoo Lee, Peter Gleckler, Kristin Chang, and Travis A. O'Brien
Geosci. Model Dev., 18, 961–976, https://doi.org/10.5194/gmd-18-961-2025, https://doi.org/10.5194/gmd-18-961-2025, 2025
Short summary
Short summary
A metrics package designed for easy analysis of atmospheric river (AR) characteristics and statistics is presented. The tool is efficient for diagnosing systematic AR bias in climate models and useful for evaluating new AR characteristics in model simulations. In climate models, landfalling AR precipitation shows dry biases globally, and AR tracks are farther poleward (equatorward) in the North and South Atlantic (South Pacific and Indian Ocean).
Mark D. Zelinka, Li-Wei Chao, Timothy A. Myers, Yi Qin, and Stephen A. Klein
Atmos. Chem. Phys., 25, 1477–1495, https://doi.org/10.5194/acp-25-1477-2025, https://doi.org/10.5194/acp-25-1477-2025, 2025
Short summary
Short summary
Clouds lie at the heart of uncertainty in both climate sensitivity and radiative forcing, making it imperative to properly diagnose their radiative effects. Here we provide a recommended methodology and code base for the community to use in performing such diagnoses using cloud radiative kernels. We show that properly accounting for changes in obscuration of lower-level clouds by upper-level clouds is important for accurate diagnosis and attribution of cloud feedbacks and adjustments.
Paul J. Durack, Karl E. Taylor, Peter J. Gleckler, Gerald A. Meehl, Bryan N. Lawrence, Curt Covey, Ronald J. Stouffer, Guillaume Levavasseur, Atef Ben-Nasser, Sebastien Denvil, Martina Stockhause, Jonathan M. Gregory, Martin Juckes, Sasha K. Ames, Fabrizio Antonio, David C. Bader, John P. Dunne, Daniel Ellis, Veronika Eyring, Sandro L. Fiore, Sylvie Joussaume, Philip Kershaw, Jean-Francois Lamarque, Michael Lautenschlager, Jiwoo Lee, Chris F. Mauzey, Matthew Mizielinski, Paola Nassisi, Alessandra Nuzzo, Eleanor O’Rourke, Jeffrey Painter, Gerald L. Potter, Sven Rodriguez, and Dean N. Williams
EGUsphere, https://doi.org/10.5194/egusphere-2024-3729, https://doi.org/10.5194/egusphere-2024-3729, 2025
Short summary
Short summary
CMIP6 was the most expansive and ambitious Model Intercomparison Project (MIP), the latest in a history, extending four decades. CMIP engaged a growing community focused on improving climate understanding, and quantifying and attributing observed climate change being experienced today. The project's profound impact is due to the combining the latest climate science and technology, enabling the latest-generation climate simulations and increasing community attention in every successive phase.
John Patrick Dunne, Helene T. Hewitt, Julie Arblaster, Frédéric Bonou, Olivier Boucher, Tereza Cavazos, Paul J. Durack, Birgit Hassler, Martin Juckes, Tomoki Miyakawa, Matthew Mizielinski, Vaishali Naik, Zebedee Nicholls, Eleanor O’Rourke, Robert Pincus, Benjamin M. Sanderson, Isla R. Simpson, and Karl E. Taylor
EGUsphere, https://doi.org/10.5194/egusphere-2024-3874, https://doi.org/10.5194/egusphere-2024-3874, 2024
Short summary
Short summary
This manuscript provides the motivation and experimental design for the seventh phase of the Coupled Model Intercomparison Project (CMIP7) to coordinate community based efforts to answer key and timely climate science questions and facilitate delivery of relevant multi-model simulations for: prediction and projection, characterization, attribution and process understanding; vulnerability, impacts and adaptations analysis; national and international climate assessments; and society at large.
Andrew C. Ross, Charles A. Stock, Vimal Koul, Thomas L. Delworth, Feiyu Lu, Andrew Wittenberg, and Michael A. Alexander
Ocean Sci., 20, 1631–1656, https://doi.org/10.5194/os-20-1631-2024, https://doi.org/10.5194/os-20-1631-2024, 2024
Short summary
Short summary
In this paper, we use a high-resolution regional ocean model to downscale seasonal ocean forecasts from the Seamless System for Prediction and EArth System Research (SPEAR) model of the Geophysical Fluid Dynamics Laboratory (GFDL). We find that the downscaled model has significantly higher prediction skill in many cases.
Seung H. Baek, Paul A. Ullrich, Bo Dong, and Jiwoo Lee
Geosci. Model Dev., 17, 8665–8681, https://doi.org/10.5194/gmd-17-8665-2024, https://doi.org/10.5194/gmd-17-8665-2024, 2024
Short summary
Short summary
We evaluate downscaled products by examining locally relevant co-variances during precipitation events. Common statistical downscaling techniques preserve expected co-variances during convective precipitation (a stationary phenomenon). However, they dampen future intensification of frontal precipitation (a non-stationary phenomenon) captured in global climate models and dynamical downscaling. Our study quantifies a ramification of the stationarity assumption underlying statistical downscaling.
Angeline G. Pendergrass, Michael P. Byrne, Oliver Watt-Meyer, Penelope Maher, and Mark J. Webb
Geosci. Model Dev., 17, 6365–6378, https://doi.org/10.5194/gmd-17-6365-2024, https://doi.org/10.5194/gmd-17-6365-2024, 2024
Short summary
Short summary
The width of the tropical rain belt affects many aspects of our climate, yet we do not understand what controls it. To better understand it, we present a method to change it in numerical model experiments. We show that the method works well in four different models. The behavior of the width is unexpectedly simple in some ways, such as how strong the winds are as it changes, but in other ways, it is more complicated, especially how temperature increases with carbon dioxide.
Jishi Zhang, Peter Bogenschutz, Qi Tang, Philip Cameron-smith, and Chengzhu Zhang
Geosci. Model Dev., 17, 3687–3731, https://doi.org/10.5194/gmd-17-3687-2024, https://doi.org/10.5194/gmd-17-3687-2024, 2024
Short summary
Short summary
We developed a regionally refined climate model that allows resolved convection and performed a 20-year projection to the end of the century. The model has a resolution of 3.25 km in California, which allows us to predict climate with unprecedented accuracy, and a resolution of 100 km for the rest of the globe to achieve efficient, self-consistent simulations. The model produces superior results in reproducing climate patterns over California that typical modern climate models cannot resolve.
John T. Fasullo, Jean-Christophe Golaz, Julie M. Caron, Nan Rosenbloom, Gerald A. Meehl, Warren Strand, Sasha Glanville, Samantha Stevenson, Maria Molina, Christine A. Shields, Chengzhu Zhang, James Benedict, Hailong Wang, and Tony Bartoletti
Earth Syst. Dynam., 15, 367–386, https://doi.org/10.5194/esd-15-367-2024, https://doi.org/10.5194/esd-15-367-2024, 2024
Short summary
Short summary
Climate model large ensembles provide a unique and invaluable means for estimating the climate response to external forcing agents and quantify contrasts in model structure. Here, an overview of the Energy Exascale Earth System Model (E3SM) version 2 large ensemble is given along with comparisons to large ensembles from E3SM version 1 and versions 1 and 2 of the Community Earth System Model. The paper provides broad and important context for users of these ensembles.
Justin L. Willson, Kevin A. Reed, Christiane Jablonowski, James Kent, Peter H. Lauritzen, Ramachandran Nair, Mark A. Taylor, Paul A. Ullrich, Colin M. Zarzycki, David M. Hall, Don Dazlich, Ross Heikes, Celal Konor, David Randall, Thomas Dubos, Yann Meurdesoif, Xi Chen, Lucas Harris, Christian Kühnlein, Vivian Lee, Abdessamad Qaddouri, Claude Girard, Marco Giorgetta, Daniel Reinert, Hiroaki Miura, Tomoki Ohno, and Ryuji Yoshida
Geosci. Model Dev., 17, 2493–2507, https://doi.org/10.5194/gmd-17-2493-2024, https://doi.org/10.5194/gmd-17-2493-2024, 2024
Short summary
Short summary
Accurate simulation of tropical cyclones (TCs) is essential to understanding their behavior in a changing climate. One way this is accomplished is through model intercomparison projects, where results from multiple climate models are analyzed to provide benchmark solutions for the wider climate modeling community. This study describes and analyzes the previously developed TC test case for nine climate models in an intercomparison project, providing solutions that aid in model development.
Lele Shu, Paul Ullrich, Xianhong Meng, Christopher Duffy, Hao Chen, and Zhaoguo Li
Geosci. Model Dev., 17, 497–527, https://doi.org/10.5194/gmd-17-497-2024, https://doi.org/10.5194/gmd-17-497-2024, 2024
Short summary
Short summary
Our team developed rSHUD v2.0, a toolkit that simplifies the use of the SHUD, a model simulating water movement in the environment. We demonstrated its effectiveness in two watersheds, one in the USA and one in China. The toolkit also facilitated the creation of the Global Hydrological Data Cloud, a platform for automatic data processing and model deployment, marking a significant advancement in hydrological research.
Karl E. Taylor
Geosci. Model Dev., 17, 415–430, https://doi.org/10.5194/gmd-17-415-2024, https://doi.org/10.5194/gmd-17-415-2024, 2024
Short summary
Short summary
Remapping gridded data in a way that preserves the conservative properties of the climate system can be essential in coupling model components and for accurate assessment of the system’s energy and mass constituents. Remapping packages capable of handling a wide variety of grids can, for some common grids, calculate remapping weights that are somewhat inaccurate. Correcting for these errors, guidelines are provided to ensure conservation when the weights are used in practice.
Mark D. Zelinka, Christopher J. Smith, Yi Qin, and Karl E. Taylor
Atmos. Chem. Phys., 23, 8879–8898, https://doi.org/10.5194/acp-23-8879-2023, https://doi.org/10.5194/acp-23-8879-2023, 2023
Short summary
Short summary
The primary uncertainty in how strongly Earth's climate has been perturbed by human activities comes from the unknown radiative impact of aerosol changes. Accurately quantifying these forcings – and their sub-components – in climate models is crucial for understanding the past and future simulated climate. In this study we describe biases in previously published estimates of aerosol radiative forcing in climate models and provide corrected estimates along with code for users to compute them.
Min-Seop Ahn, Paul A. Ullrich, Peter J. Gleckler, Jiwoo Lee, Ana C. Ordonez, and Angeline G. Pendergrass
Geosci. Model Dev., 16, 3927–3951, https://doi.org/10.5194/gmd-16-3927-2023, https://doi.org/10.5194/gmd-16-3927-2023, 2023
Short summary
Short summary
We introduce a framework for regional-scale evaluation of simulated precipitation distributions with 62 climate reference regions and 10 metrics and apply it to evaluate CMIP5 and CMIP6 models against multiple satellite-based precipitation products. The common model biases identified in this study are mainly associated with the overestimated light precipitation and underestimated heavy precipitation. These biases persist from earlier-generation models and have been slightly improved in CMIP6.
Qi Tang, Jean-Christophe Golaz, Luke P. Van Roekel, Mark A. Taylor, Wuyin Lin, Benjamin R. Hillman, Paul A. Ullrich, Andrew M. Bradley, Oksana Guba, Jonathan D. Wolfe, Tian Zhou, Kai Zhang, Xue Zheng, Yunyan Zhang, Meng Zhang, Mingxuan Wu, Hailong Wang, Cheng Tao, Balwinder Singh, Alan M. Rhoades, Yi Qin, Hong-Yi Li, Yan Feng, Yuying Zhang, Chengzhu Zhang, Charles S. Zender, Shaocheng Xie, Erika L. Roesler, Andrew F. Roberts, Azamat Mametjanov, Mathew E. Maltrud, Noel D. Keen, Robert L. Jacob, Christiane Jablonowski, Owen K. Hughes, Ryan M. Forsyth, Alan V. Di Vittorio, Peter M. Caldwell, Gautam Bisht, Renata B. McCoy, L. Ruby Leung, and David C. Bader
Geosci. Model Dev., 16, 3953–3995, https://doi.org/10.5194/gmd-16-3953-2023, https://doi.org/10.5194/gmd-16-3953-2023, 2023
Short summary
Short summary
High-resolution simulations are superior to low-resolution ones in capturing regional climate changes and climate extremes. However, uniformly reducing the grid size of a global Earth system model is too computationally expensive. We provide an overview of the fully coupled regionally refined model (RRM) of E3SMv2 and document a first-of-its-kind set of climate production simulations using RRM at an economic cost. The key to this success is our innovative hybrid time step method.
Abhishekh Kumar Srivastava, Paul Aaron Ullrich, Deeksha Rastogi, Pouya Vahmani, Andrew Jones, and Richard Grotjahn
Geosci. Model Dev., 16, 3699–3722, https://doi.org/10.5194/gmd-16-3699-2023, https://doi.org/10.5194/gmd-16-3699-2023, 2023
Short summary
Short summary
Stakeholders need high-resolution regional climate data for applications such as assessing water availability and mountain snowpack. This study examines 3 h and 24 h historical precipitation over the contiguous United States in the 12 km WRF version 4.2.1-based dynamical downscaling of the ERA5 reanalysis. WRF improves precipitation characteristics such as the annual cycle and distribution of the precipitation maxima, but it also displays regionally and seasonally varying precipitation biases.
Zeyu Xue, Paul Ullrich, and Lai-Yung Ruby Leung
Hydrol. Earth Syst. Sci., 27, 1909–1927, https://doi.org/10.5194/hess-27-1909-2023, https://doi.org/10.5194/hess-27-1909-2023, 2023
Short summary
Short summary
We examine the sensitivity and robustness of conclusions drawn from the PGW method over the NEUS by conducting multiple PGW experiments and varying the perturbation spatial scales and choice of perturbed meteorological variables to provide a guideline for this increasingly popular regional modeling method. Overall, we recommend PGW experiments be performed with perturbations to temperature or the combination of temperature and wind at the gridpoint scale, depending on the research question.
David H. Marsico and Paul A. Ullrich
Geosci. Model Dev., 16, 1537–1551, https://doi.org/10.5194/gmd-16-1537-2023, https://doi.org/10.5194/gmd-16-1537-2023, 2023
Short summary
Short summary
Climate models involve several different components, such as the atmosphere, ocean, and land models. Information needs to be exchanged, or remapped, between these models, and devising algorithms for performing this exchange is important for ensuring the accuracy of climate simulations. In this paper, we examine the efficacy of several traditional and novel approaches to remapping on the sphere and demonstrate where our approaches offer improvement.
Iris Elisabeth de Vries, Sebastian Sippel, Angeline Greene Pendergrass, and Reto Knutti
Earth Syst. Dynam., 14, 81–100, https://doi.org/10.5194/esd-14-81-2023, https://doi.org/10.5194/esd-14-81-2023, 2023
Short summary
Short summary
Precipitation change is an important consequence of climate change, but it is hard to detect and quantify. Our intuitive method yields robust and interpretable detection of forced precipitation change in three observational datasets for global mean and extreme precipitation, but the different observational datasets show different magnitudes of forced change. Assessment and reduction of uncertainties surrounding forced precipitation change are important for future projections and adaptation.
Chengzhu Zhang, Jean-Christophe Golaz, Ryan Forsyth, Tom Vo, Shaocheng Xie, Zeshawn Shaheen, Gerald L. Potter, Xylar S. Asay-Davis, Charles S. Zender, Wuyin Lin, Chih-Chieh Chen, Chris R. Terai, Salil Mahajan, Tian Zhou, Karthik Balaguru, Qi Tang, Cheng Tao, Yuying Zhang, Todd Emmenegger, Susannah Burrows, and Paul A. Ullrich
Geosci. Model Dev., 15, 9031–9056, https://doi.org/10.5194/gmd-15-9031-2022, https://doi.org/10.5194/gmd-15-9031-2022, 2022
Short summary
Short summary
Earth system model (ESM) developers run automated analysis tools on data from candidate models to inform model development. This paper introduces a new Python package, E3SM Diags, that has been developed to support ESM development and use routinely in the development of DOE's Energy Exascale Earth System Model. This tool covers a set of essential diagnostics to evaluate the mean physical climate from simulations, as well as several process-oriented and phenomenon-based evaluation diagnostics.
Sjoukje Y. Philip, Sarah F. Kew, Geert Jan van Oldenborgh, Faron S. Anslow, Sonia I. Seneviratne, Robert Vautard, Dim Coumou, Kristie L. Ebi, Julie Arrighi, Roop Singh, Maarten van Aalst, Carolina Pereira Marghidan, Michael Wehner, Wenchang Yang, Sihan Li, Dominik L. Schumacher, Mathias Hauser, Rémy Bonnet, Linh N. Luu, Flavio Lehner, Nathan Gillett, Jordis S. Tradowsky, Gabriel A. Vecchi, Chris Rodell, Roland B. Stull, Rosie Howard, and Friederike E. L. Otto
Earth Syst. Dynam., 13, 1689–1713, https://doi.org/10.5194/esd-13-1689-2022, https://doi.org/10.5194/esd-13-1689-2022, 2022
Short summary
Short summary
In June 2021, the Pacific Northwest of the US and Canada saw record temperatures far exceeding those previously observed. This attribution study found such a severe heat wave would have been virtually impossible without human-induced climate change. Assuming no nonlinear interactions, such events have become at least 150 times more common, are about 2 °C hotter and will become even more common as warming continues. Therefore, adaptation and mitigation are urgently needed to prepare society.
Yona Silvy, Clément Rousset, Eric Guilyardi, Jean-Baptiste Sallée, Juliette Mignot, Christian Ethé, and Gurvan Madec
Geosci. Model Dev., 15, 7683–7713, https://doi.org/10.5194/gmd-15-7683-2022, https://doi.org/10.5194/gmd-15-7683-2022, 2022
Short summary
Short summary
A modeling framework is introduced to understand and decompose the mechanisms causing the ocean temperature, salinity and circulation to change since the pre-industrial period and into 21st century scenarios of global warming. This framework aims to look at the response to changes in the winds and in heat and freshwater exchanges at the ocean interface in global climate models, throughout the 1850–2100 period, to unravel their individual effects on the changing physical structure of the ocean.
Vijay S. Mahadevan, Jorge E. Guerra, Xiangmin Jiao, Paul Kuberry, Yipeng Li, Paul Ullrich, David Marsico, Robert Jacob, Pavel Bochev, and Philip Jones
Geosci. Model Dev., 15, 6601–6635, https://doi.org/10.5194/gmd-15-6601-2022, https://doi.org/10.5194/gmd-15-6601-2022, 2022
Short summary
Short summary
Coupled Earth system models require transfer of field data between multiple components with varying spatial resolutions to determine the correct climate behavior. We present the Metrics for Intercomparison of Remapping Algorithms (MIRA) protocol to evaluate the accuracy, conservation properties, monotonicity, and local feature preservation of four different remapper algorithms for various unstructured mesh problems of interest. Future extensions to more practical use cases are also discussed.
Amy H. Peace, Ben B. B. Booth, Leighton A. Regayre, Ken S. Carslaw, David M. H. Sexton, Céline J. W. Bonfils, and John W. Rostron
Earth Syst. Dynam., 13, 1215–1232, https://doi.org/10.5194/esd-13-1215-2022, https://doi.org/10.5194/esd-13-1215-2022, 2022
Short summary
Short summary
Anthropogenic aerosol emissions have been linked to driving climate responses such as shifts in the location of tropical rainfall. However, the interaction of aerosols with climate remains one of the most uncertain aspects of climate modelling and limits our ability to predict future climate change. We use an ensemble of climate model simulations to investigate what impact the large uncertainty in how aerosols interact with climate has on predicting future tropical rainfall shifts.
Pradeebane Vaittinada Ayar, Laurent Bopp, Jim R. Christian, Tatiana Ilyina, John P. Krasting, Roland Séférian, Hiroyuki Tsujino, Michio Watanabe, Andrew Yool, and Jerry Tjiputra
Earth Syst. Dynam., 13, 1097–1118, https://doi.org/10.5194/esd-13-1097-2022, https://doi.org/10.5194/esd-13-1097-2022, 2022
Short summary
Short summary
The El Niño–Southern Oscillation is the main driver for the natural variability of global atmospheric CO2. It modulates the CO2 fluxes in the tropical Pacific with anomalous CO2 influx during El Niño and outflux during La Niña. This relationship is projected to reverse by half of Earth system models studied here under the business-as-usual scenario. This study shows models that simulate a positive bias in surface carbonate concentrations simulate a shift in the ENSO–CO2 flux relationship.
Po-Lun Ma, Bryce E. Harrop, Vincent E. Larson, Richard B. Neale, Andrew Gettelman, Hugh Morrison, Hailong Wang, Kai Zhang, Stephen A. Klein, Mark D. Zelinka, Yuying Zhang, Yun Qian, Jin-Ho Yoon, Christopher R. Jones, Meng Huang, Sheng-Lun Tai, Balwinder Singh, Peter A. Bogenschutz, Xue Zheng, Wuyin Lin, Johannes Quaas, Hélène Chepfer, Michael A. Brunke, Xubin Zeng, Johannes Mülmenstädt, Samson Hagos, Zhibo Zhang, Hua Song, Xiaohong Liu, Michael S. Pritchard, Hui Wan, Jingyu Wang, Qi Tang, Peter M. Caldwell, Jiwen Fan, Larry K. Berg, Jerome D. Fast, Mark A. Taylor, Jean-Christophe Golaz, Shaocheng Xie, Philip J. Rasch, and L. Ruby Leung
Geosci. Model Dev., 15, 2881–2916, https://doi.org/10.5194/gmd-15-2881-2022, https://doi.org/10.5194/gmd-15-2881-2022, 2022
Short summary
Short summary
An alternative set of parameters for E3SM Atmospheric Model version 1 has been developed based on a tuning strategy that focuses on clouds. When clouds in every regime are improved, other aspects of the model are also improved, even though they are not the direct targets for calibration. The recalibrated model shows a lower sensitivity to anthropogenic aerosols and surface warming, suggesting potential improvements to the simulated climate in the past and future.
Claudia Tebaldi, Kalyn Dorheim, Michael Wehner, and Ruby Leung
Earth Syst. Dynam., 12, 1427–1501, https://doi.org/10.5194/esd-12-1427-2021, https://doi.org/10.5194/esd-12-1427-2021, 2021
Short summary
Short summary
We address the question of how large an initial condition ensemble of climate model simulations should be if we are concerned with accurately projecting future changes in temperature and precipitation extremes. We find that for most cases (and both models considered), an ensemble of 20–25 members is sufficient for many extreme metrics, spatial scales and time horizons. This may leave computational resources to tackle other uncertainties in climate model simulations with our ensembles.
Trevor J. McDougall, Paul M. Barker, Ryan M. Holmes, Rich Pawlowicz, Stephen M. Griffies, and Paul J. Durack
Geosci. Model Dev., 14, 6445–6466, https://doi.org/10.5194/gmd-14-6445-2021, https://doi.org/10.5194/gmd-14-6445-2021, 2021
Short summary
Short summary
We show that the way that the air–sea heat flux is treated in ocean models means that the model's temperature variable should be interpreted as being Conservative Temperature, irrespective of whether the equation of state used in an ocean model is EOS-80 or TEOS-10.
Benjamin M. Sanderson, Angeline G. Pendergrass, Charles D. Koven, Florent Brient, Ben B. B. Booth, Rosie A. Fisher, and Reto Knutti
Earth Syst. Dynam., 12, 899–918, https://doi.org/10.5194/esd-12-899-2021, https://doi.org/10.5194/esd-12-899-2021, 2021
Short summary
Short summary
Emergent constraints promise a pathway to the reduction in climate projection uncertainties by exploiting ensemble relationships between observable quantities and unknown climate response parameters. This study considers the robustness of these relationships in light of biases and common simplifications that may be present in the original ensemble of climate simulations. We propose a classification scheme for constraints and a number of practical case studies.
Paul A. Ullrich, Colin M. Zarzycki, Elizabeth E. McClenny, Marielle C. Pinheiro, Alyssa M. Stansfield, and Kevin A. Reed
Geosci. Model Dev., 14, 5023–5048, https://doi.org/10.5194/gmd-14-5023-2021, https://doi.org/10.5194/gmd-14-5023-2021, 2021
Short summary
Short summary
TempestExtremes (TE) is a multifaceted framework for feature detection, tracking, and scientific analysis of regional or global Earth system datasets. Version 2.1 of TE now provides extensive support for nodal and areal features. This paper describes the algorithms that have been added to the TE framework since version 1.0 and gives several examples of how these can be combined to produce composite algorithms for evaluating and understanding atmospheric features.
Christina Heinze-Deml, Sebastian Sippel, Angeline G. Pendergrass, Flavio Lehner, and Nicolai Meinshausen
Geosci. Model Dev., 14, 4977–4999, https://doi.org/10.5194/gmd-14-4977-2021, https://doi.org/10.5194/gmd-14-4977-2021, 2021
Short summary
Short summary
Quantifying dynamical and thermodynamical components of regional precipitation change is a key challenge in climate science. We introduce a novel statistical model (Latent Linear Adjustment Autoencoder) that combines the flexibility of deep neural networks with the robustness advantages of linear regression. The method enables estimation of the contribution of a coarse-scale atmospheric circulation proxy to daily precipitation at high resolution and in a spatially coherent manner.
Prabhat, Karthik Kashinath, Mayur Mudigonda, Sol Kim, Lukas Kapp-Schwoerer, Andre Graubner, Ege Karaismailoglu, Leo von Kleist, Thorsten Kurth, Annette Greiner, Ankur Mahesh, Kevin Yang, Colby Lewis, Jiayi Chen, Andrew Lou, Sathyavat Chandran, Ben Toms, Will Chapman, Katherine Dagon, Christine A. Shields, Travis O'Brien, Michael Wehner, and William Collins
Geosci. Model Dev., 14, 107–124, https://doi.org/10.5194/gmd-14-107-2021, https://doi.org/10.5194/gmd-14-107-2021, 2021
Short summary
Short summary
Detecting extreme weather events is a crucial step in understanding how they change due to climate change. Deep learning (DL) is remarkable at pattern recognition; however, it works best only when labeled datasets are available. We create
ClimateNet– an expert-labeled curated dataset – to train a DL model for detecting weather events and predicting changes in extreme precipitation. This work paves the way for DL-based automated, high-fidelity, and highly precise analytics of climate data.
Hsi-Yen Ma, Chen Zhou, Yunyan Zhang, Stephen A. Klein, Mark D. Zelinka, Xue Zheng, Shaocheng Xie, Wei-Ting Chen, and Chien-Ming Wu
Geosci. Model Dev., 14, 73–90, https://doi.org/10.5194/gmd-14-73-2021, https://doi.org/10.5194/gmd-14-73-2021, 2021
Short summary
Short summary
We propose an experimental design of a suite of multi-year, short-term hindcasts and compare them with corresponding observations or measurements for periods based on different weather and climate phenomena. This atypical way of evaluating model performance is particularly useful and beneficial, as these hindcasts can give scientists a robust picture of modeled precipitation, and cloud and radiation processes from their diurnal variation to year-to-year variability.
Lukas Brunner, Angeline G. Pendergrass, Flavio Lehner, Anna L. Merrifield, Ruth Lorenz, and Reto Knutti
Earth Syst. Dynam., 11, 995–1012, https://doi.org/10.5194/esd-11-995-2020, https://doi.org/10.5194/esd-11-995-2020, 2020
Short summary
Short summary
In this study, we weight climate models by their performance with respect to simulating aspects of historical climate and their degree of interdependence. Our method is found to increase projection skill and to correct for structurally similar models. The weighted end-of-century mean warming (2081–2100 relative to 1995–2014) is 3.7 °C with a likely (66 %) range of 3.1 to 4.6 °C for the strong climate change scenario SSP5-8.5; this is a reduction of 0.4 °C compared with the unweighted mean.
Landon A. Rieger, Jason N. S. Cole, John C. Fyfe, Stephen Po-Chedley, Philip J. Cameron-Smith, Paul J. Durack, Nathan P. Gillett, and Qi Tang
Geosci. Model Dev., 13, 4831–4843, https://doi.org/10.5194/gmd-13-4831-2020, https://doi.org/10.5194/gmd-13-4831-2020, 2020
Short summary
Short summary
Recently, the stratospheric aerosol forcing dataset used as an input to the Coupled Model Intercomparison Project phase 6 was updated. This work explores the impact of those changes on the modelled historical climates in the CanESM5 and EAMv1 models. Temperature differences in the stratosphere shortly after the Pinatubo eruption are found to be significant, but surface temperatures and precipitation do not show a significant change.
Mark D. Risser and Michael F. Wehner
Adv. Stat. Clim. Meteorol. Oceanogr., 6, 115–139, https://doi.org/10.5194/ascmo-6-115-2020, https://doi.org/10.5194/ascmo-6-115-2020, 2020
Short summary
Short summary
Evaluation of modern high-resolution global climate models often does not account for the geographic location of the underlying weather station data. In this paper, we quantify the impact of geographic sampling on the relative performance of climate model representations of precipitation extremes over the United States. We find that properly accounting for the geographic sampling of weather stations can significantly change the assessment of model performance.
Cited articles
Adler, R. F., Sapiano, M. R., Huffman, G. J., Wang, J. J., Gu, G., Bolvin, D., Chiu, L., Schneider, U., Becker, A., Nelkin, E., Xie, P., Ferraro, R., and Shin, D.-B.: The Global Precipitation Climatology Project (GPCP) monthly analysis (new version 2.3) and a review of 2017 global precipitation. Atmosphere, 9, 138, https://doi.org/10.3390/atmos9040138, 2018.
Ahn, M.-S., Kim, D. H., Sperber, K. R., Kang, I.-S., Maloney, E. D., Waliser, D. E., and Hendon, H. H.: MJO simulation in CMIP5 climate models: MJO skill metrics and process-oriented diagnosis, Clim. Dynam., 49, 4023–4045, https://doi.org/10.1007/s00382-017-3558-4, 2017.
Ahn, M.-S., Gleckler, P. J., Lee, J., Pendergrass, A. G., and Jakob, C.: Benchmarking Simulated Precipitation Variability Amplitude across Time Scales, J. Climate, 35, 3173–3196, https://doi.org/10.1175/jcli-d-21-0542.1, 2022.
Ahn, M.-S., Ullrich, P. A., Gleckler, P. J., Lee, J., Ordonez, A. C., and Pendergrass, A. G.: Evaluating precipitation distributions at regional scales: a benchmarking framework and application to CMIP5 and 6 models, Geosci. Model Dev., 16, 3927–3951, https://doi.org/10.5194/gmd-16-3927-2023, 2023.
Anaconda pcmdi_metrics: https://anaconda.org/conda-forge/pcmdi_metrics, last access: 8 May 2024.
Arcodia, M., Barnes, E. A., Mayer, K., Lee, J., Ordonez, A., and Ahn, M.-S.: Assessing decadal variability of subseasonal forecasts of opportunity using explainable AI, Environ. Res., 2, 045002, https://doi.org/10.1088/2752-5295/aced60, 2023.
Ashfaq, M., Rastogi, D., Kitson, J., Abid, M. A., and Kao, S.-C.: Evaluation of CMIP6 GCMs over the CONUS for downscaling studies, J. Geophys. Res.-Atmos., 127, e2022JD036659, https://doi.org/10.1029/2022JD036659, 2022.
Bayr, T., Wengel, C., Latif, M., Dommenget, D., Lübbecke, J., and Park, W.: Error compensation of ENSO atmospheric feedbacks in climate models and its influence on simulated ENSO dynamics, Clim. Dynam., 53, 155–172, https://doi.org/10.1007/s00382-018-4575-7, 2019.
Biard, J. C. and Kunkel, K. E.: Automated detection of weather fronts using a deep learning neural network, Adv. Stat. Clim. Meteorol. Oceanogr., 5, 147–160, https://doi.org/10.5194/ascmo-5-147-2019, 2019.
Bellenger, H., Guilyardi, E., Leloup, J., Lengaigne, M., and Vialard, J.: ENSO representation in climate models: from CMIP3 to CMIP5, Clim. Dynam., 42, 1999–2018, https://doi.org/10.1007/s00382-013-1783-z, 2014.
Boucher, O., Servonnat, J., Albright, A. L., Aumont, O., Balkanski, Y., Bastrikov, V., Bekki, S., Bonnet, R., Bony, S., Bopp, L., Braconnot, P., Brockmann, P., Cadule, P., Caubel, A., Cheruy, F., Codron, F., Cozic, A., Cugnet, D., D'Andrea, F., Davini, P., De Lavergne, C., Denvil, S., Deshayes, J., Devilliers, M., Ducharne, A., Dufresne, J. L., Dupont, E., Ethé, C., Fairhead, L., Falletti, L., Flavoni, S., Foujols, M. A., Gardoll, S., Gastineau, G., Ghattas, J., Grandpeix, J. Y., Guenet, B., Lionel, E. G., Guilyardi, E., Guimberteau, M., Hauglustaine, D., Hourdin, F., Idelkadi, A., Joussaume, S., Kageyama, M., Khodri, M., Krinner, G., Lebas, N., Levavasseur, G., Lévy, C., Li, L., Lott, F., Lurton, T., Luyssaert, S., Madec, G., Madeleine, J.-B., Maignan, F., Marchand, M., Marti, O., Mellul, L., Meurdesoif, Y., Mignot, J., Musat, I., Ottlé, C., Peylin, P., Planton, Y., Polcher, J., Rio, C., Rochetin, N., Rousset, C., Sepulchre, P., Sima, A., Swingedouw, D., Thiéblemont, R., Traore, A. K., Vancoppenolle, M., Vial, J., Vialard, J., Viovy, N., and Vuichard, N.: Presentation and evaluation of the IPSL-CM6A-LR Climate Model, J. Adv. Model. Earth Sy., 12, e2019MS002010, https://doi.org/10.1029/2019ms002010, 2020.
Caldwell, P., Mametjanov, A., Tang, Q., Van Roekel, L., Golaz, J.-C., Lin, W., Bader, D. C., Keen, N. D., Feng, Y., Jacob, R., Maltrud, M., Roberts, A., Taylor, M. A., Veneziani, M., Wang, H., Wolfe, J. D., Balaguru, K., Cameron-Smith, P. J., Dong, L., Klein, S. A., Leung, L. R., Li, H., Li, Q., Liu, X., Neale, R., Pinheiro, M. C., Qian, Y., Ullrich, P. A., Xie, S., Yang, Y., Zhang, Y., Zhang, K., and Zhou, T.: The DOE E3SM Coupled Model Version 1: description and results at high resolution, J. Adv. Model. Earth Sy., 11, 4095–4146, https://doi.org/10.1029/2019ms001870, 2019.
Chen, H.-C., Jin, F.-F., Zhao, S., Wittenberg, A. T., and Xie, S.: ENSO dynamics in the E3SM-1-0, CESM2, and GFDL-CM4 climate models, J. Climate, 34, 9365–9384, https://doi.org/10.1175/JCLI-D-21-0355.1, 2021.
Collier, N., Hoffman, F. M., Lawrence, D. M., Keppel-Aleks, G., Koven, C. D., Riley, W. J., Mu, M., and Randerson, J. T.: The International Land Model Benchmarking (ILAMB) System: Design, theory, and implementation, J. Adv. Model. Earth Sy., 10, 2731–2754, https://doi.org/10.1029/2018ms001354, 2018.
Covey, C., AchutaRao, K., Cubasch, U., Jones, P., Lambert, S. J., Mann, M., Phillips, T. J., and Taylor, K. E.: An overview of results from the Coupled Model Intercomparison Project, Global. Planet. Change, 37, 103–133, https://doi.org/10.1016/s0921-8181(02)00193-5, 2003.
Covey, C., Gleckler, P. J., Doutriaux, C., Williams, D. N., Dai, A., Fasullo, J. T., Trenberth, K. E., and Berg, A.: Metrics for the diurnal cycle of precipitation: toward routine benchmarks for climate models, J. Climate, 29, 4461–4471, https://doi.org/10.1175/jcli-d-15-0664.1, 2016.
Crockford, D.: The application/json media type for javascript object notation (json) (No. rfc4627), https://www.rfc-editor.org/rfc/pdfrfc/rfc4627.txt.pdf (last access: 4 April 2024), 2006.
Crockford, D. and Morningstar, C.: The JSON Data Interchange Syntax, ECMA-404, ECMA International, https://doi.org/10.13140/RG.2.2.28181.14560, 2017.
Dalelane, C., Winderlich, K., and Walter, A.: Evaluation of global teleconnections in CMIP6 climate projections using complex networks, Earth Syst. Dynam., 14, 17–37, https://doi.org/10.5194/esd-14-17-2023, 2023.
Dawson, A.: eofs: A Library for EOF Analysis of Meteorological, Oceanographic, and Climate Data, J. Open Res. Software, 4, e14, https://doi.org/10.5334/jors.122, 2016.
Dee, D. P., Uppala, S. M., Simmons, A. J., Berrisford, P., Poli, P., Kobayashi, S., Andrae, U., Balmaseda, M. A., Balsamo, G., Bauer, P., Bechtold, P., Beljaars, A. C. M., van de Berg, L., Bidlot, J., Bormann, N., Delsol, C., Dragani, R., Fuentes, M., Geer, A. J., Haimberger, L., Healy, S. B., Hersbach, H., Hólm, E. V., Isaksen, L., Kållberg, P., Köhler, M., Matricardi, M., McNally, A. P., Monge-Sanz, B. M., Morcrette, J.-J., Park, B.-K., Peubey, C., de Rosnay, P., Tavolato, C., Thépaut, J.-N., and Vitart, F.: The ERA-Interim reanalysis: Configuration and performance of the data assimilation system, Q. J. Roy. Meteor. Soc., 137, 553–597, https://doi.org/10.1002/qj.828, 2011.
Deser, C. and Phillips, A. S.: Defining the internal component of Atlantic multidecadal variability in a changing climate, Geophys. Res. Lett., 48, e2021GL095023, https://doi.org/10.1029/2021gl095023, 2021.
Doutriaux, C., Nadeau, D., Wittenburg, S., Lipsa, D., Muryanto, L., Chaudhary, A., and Williams, D. N.: CDAT/cdat: CDAT 8.1, Zenodo [code], https://doi.org/10.5281/zenodo.2586088, 2019.
Durack, P. J., Taylor, K. E., Eyring, V., Ames, S., Hoang, T., Nadeau, D., Doutriaux, C., Stockhause, M., and Gleckler, P. J.: Toward standardized data sets for climate model experimentation, Eos T. Am. Geophys. Un., 99, https://doi.org/10.1029/2018eo101751, 2018.
Eaton, B., Gregory, J., Drach, B., Taylor, K., Hankin, S., Blower, J., Caron, J., Signell, R., Bentley, P., Rappa, G., Höck, H., Pamment, A., Juckes, M., Raspaud, M., Horne, R., Whiteaker, T., Blodgett, D., Zender, C., Lee, D., Hassell, D., Snow, A. D., Kölling, T., Allured, D., Jelenak, A., Soerensen, A. M., Gaultier, L., and Herlédan, S.: NetCDF Climate and Forecast (CF) Meta-data Conventions V1.10, http://cfconventions.org/Data/cf-conventions/cf-conventions-1.10/cf-conventions.html (last access: 4 April 2024), 2022.
ESGF LLNL Metagrid: https://esgf-node.llnl.gov/, last access: 8 May 2024.
Eyring, V., Righi, M., Lauer, A., Evaldsson, M., Wenzel, S., Jones, C., Anav, A., Andrews, O., Cionni, I., Davin, E. L., Deser, C., Ehbrecht, C., Friedlingstein, P., Gleckler, P., Gottschaldt, K.-D., Hagemann, S., Juckes, M., Kindermann, S., Krasting, J., Kunert, D., Levine, R., Loew, A., Mäkelä, J., Martin, G., Mason, E., Phillips, A. S., Read, S., Rio, C., Roehrig, R., Senftleben, D., Sterl, A., van Ulft, L. H., Walton, J., Wang, S., and Williams, K. D.: ESMValTool (v1.0) – a community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP, Geosci. Model Dev., 9, 1747–1802, https://doi.org/10.5194/gmd-9-1747-2016, 2016a.
Eyring, V., Bony, S., Meehl, G. A., Senior, C. A., Stevens, B., Stouffer, R. J., and Taylor, K. E.: Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization, Geosci. Model Dev., 9, 1937–1958, https://doi.org/10.5194/gmd-9-1937-2016, 2016b.
Eyring, V., Cox, P. M., Flato, G. M., Gleckler, P. J., Abramowitz, G., Caldwell, P., Collins, W. D., Gier, B. K., Hall, A., Hoffman, F. M., Hurtt, G. C., Jahn, A., Jones, C. D., Klein, S. A., Krasting, J. P., Kwiatkowski, L., Lorenz, R., Maloney, E. D., Meehl, G. A., Pendergrass, A. G., Pincus, R., Ruane, A. C., Russell, J. L., Sanderson, B. M., Santer, B. D., Sherwood, S. C., Simpson, I. R., Stouffer, R. J., and Williamson, M. S.: Taking climate model evaluation to the next level, Nat. Clim. Change, 9, 102–110, https://doi.org/10.1038/s41558-018-0355-y, 2019.
Eyring, V., Bock, L., Lauer, A., Righi, M., Schlund, M., Andela, B., Arnone, E., Bellprat, O., Brötz, B., Caron, L.-P., Carvalhais, N., Cionni, I., Cortesi, N., Crezee, B., Davin, E. L., Davini, P., Debeire, K., de Mora, L., Deser, C., Docquier, D., Earnshaw, P., Ehbrecht, C., Gier, B. K., Gonzalez-Reviriego, N., Goodman, P., Hagemann, S., Hardiman, S., Hassler, B., Hunter, A., Kadow, C., Kindermann, S., Koirala, S., Koldunov, N., Lejeune, Q., Lembo, V., Lovato, T., Lucarini, V., Massonnet, F., Müller, B., Pandde, A., Pérez-Zanón, N., Phillips, A., Predoi, V., Russell, J., Sellar, A., Serva, F., Stacke, T., Swaminathan, R., Torralba, V., Vegas-Regidor, J., von Hardenberg, J., Weigel, K., and Zimmermann, K.: Earth System Model Evaluation Tool (ESMValTool) v2.0 – an extended set of large-scale diagnostics for quasi-operational and comprehensive evaluation of Earth system models in CMIP, Geosci. Model Dev., 13, 3383–3438, https://doi.org/10.5194/gmd-13-3383-2020, 2020.
Eyring, V., Gillett, N. P., Achuta Rao, K. M., Barimalala, R., Barreiro Parrillo, M., Bellouin, N., Cassou, C., Durack, P. J., Kosaka, Y., McGregor, S. and Min, S., Morgenstern, O., and Sun, Y.: Human Influence on the Climate System, in: Climate Change 2021: The Physical Science Basis, Contribution of Working Group I to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change, 105, 423–552, https://doi.org/10.1017/9781009157896.005, 2021.
Fasullo, J. T.: Evaluating simulated climate patterns from the CMIP archives using satellite and reanalysis datasets using the Climate Model Assessment Tool (CMATv1), Geosci. Model Dev., 13, 3627–3642, https://doi.org/10.5194/gmd-13-3627-2020, 2020.
Fasullo, J. T., Phillips, A. S., and Deser, C.: Evaluation of leading modes of climate variability in the CMIP archives, J. Climate, 33, 5527–5545, https://doi.org/10.1175/jcli-d-19-1024.1, 2020.
Ferraro, R., Waliser, D. E., Gleckler, P. J., Taylor, K. E., and Eyring, V.: Evolving OBS4MIPS to support Phase 6 of the Coupled Model Intercomparison Project (CMIP6), B. Am. Meteorol. Soc., 96, ES131–ES133, https://doi.org/10.1175/bams-d-14-00216.1, 2015.
Flato, G., Marotzke, J., Abiodun, B., Braconnot, P., Chou, S. C., Collins, W., Cox, P., Driouech, F., Emori, S., Eyring, V., and Forest, C.: Evaluation of climate models, in: Climate change 2013: the physical science basis, Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change, Cambridge University Press, 741–866, 2014.
Fu, W., Moore, J. K., Primeau, F., Collier, N., Ogunro, O. O., Hoffman, F. M., and Randerson, J. T.: Evaluation of ocean biogeochemistry and carbon cycling in CMIP earth system models with the international ocean model benchmarking (IOMB) software System. J. Geophys. Res.-Oceans, 127, e2022JC018965, https://doi.org/10.1029/2022JC018965, 2022.
Gates, W. L.: AN AMS continuing series: Global CHANGE–AMIP: The Atmospheric Model Intercomparison Project, B. Am. Meteorol. Soc., 73, 1962–1970, 1992.
Gates, W. L., Henderson-Sellers, A., Boer, G. J., Folland, C. K., Kitoh, A., McAvaney, B. J., Semazzi, F., Smith, N., Weaver, A. J., and Zeng, Q. C.: Climate models – evaluation, Climate Change, 1, 229–284, 1995.
Gates, W. L., Boyle, J. S., Covey, C., Dease, C. G., Doutriaux, C. M., Drach, R. S., Fiorino, M., Gleckler, P. J., Hnilo, J. J., Marlais, S. M., and Phillips, T. J.: An overview of the results of the Atmospheric Model Intercomparison Project (AMIP I), B. Am. Meteorol. Soc., 80, 29–56, 1999.
Gleckler, P. J., Taylor, K. E., and Doutriaux, C.: Performance metrics for climate models, J. Geophys. Res., 113, D06104, https://doi.org/10.1029/2007jd008972, 2008.
Gleckler, P. J., Ferraro, R., and Waliser, D. E.: Improving use of satellite data in evaluating climate models, Eos T. Am. Geophys. Un., 92, 172, https://doi.org/10.1029/2011eo200005, 2011.
Gleckler, P. J., Doutriaux, C., Durack, P. J., Taylor, K. E., Zhang, Y., Williams, D. N., Mason, E., and Servonnat, J.: A more powerful reality test for climate models, Eos T. Am. Geophys. Un., 97, https://doi.org/10.1029/2016eo051663, 2016.
Golaz, J.-C., Caldwell, P., Van Roekel, L., Petersen, M. R., Tang, Q., Wolfe, J. D., Abeshu, G. W., Anantharaj, V., Asay-Davis, X., Bader, D. C., Baldwin, S., Bisht, G., Bogenschutz, P., Branstetter, M. L., Brunke, M. A., Brus, S., Burrows, S. M., Cameron-Smith, P. J., Donahue, A. S., Deakin, M., Easter, R. C., Evans, K. J., Feng, Y., Flanner, M., Foucar, J. G., Fyke, J., Griffin, B. M., Hannay, C., Harrop, B. E., Hoffman, M. J., Hunke, E., Jacob, R., Jacobsen, D. W., Jeffery, N., Jones, P. W., Keen, N. D., Klein, S. A., Larson, V. E., Leung, L. R., Li, H. Y., Lin, W., Lipscomb, W. H., Lun, P., Mahajan, S., Maltrud, M., Mametjanov, A., McClean, J. L., McCoy, R., Neale, R., Price, S., Qian, Y., Rasch, P. J., Eyre, J. E. J. R., Riley, W. J., Ringler, T. D., Roberts, A., Roesler, E. L., Salinger, A. G., Shaheen, Z., Shi, X., Singh, B., Tang, J., Taylor, M. A., Thornton, P. E., Turner, A. K., Veneziani, M., Wan, H., Wang, H., Wang, S., Williams, D. N., Wolfram, P. J., Worley, P. H., Xie, S., Yang, Y., Yoon, J., Zelinka, M. D., Zender, C. S., Zeng, X., Zhang, C., Zhang, K., Zhang, Y., Zheng, X., Zhou, T., and Zhu, Q.: The DOE E3SM Coupled Model Version 1: Overview and evaluation at standard resolution, J. Adv. Model. Earth Sy., 11, 2089–2129, https://doi.org/10.1029/2018ms001603, 2019.
Goldenson, N., Leung, L. R., Mearns, L. O., Pierce, D. W., Reed, K. A., Simpson, I. R., Ullrich, P., Krantz, W., Hall, A., Jones, A., and Rahimi, S.: Use-Inspired, Process-Oriented GCM Selection: Prioritizing Models for Regional Dynamical Downscaling, B. Am. Meteorol. Soc., 104, E1619–E1629, https://doi.org/10.1175/BAMS-D-23-0100.1, 2023.
Guilyardi, E., Wittenberg, A., Fedorov, A., Collins, M., Wang, C., Capotondi, A., Van Oldenborgh, G. J., and Stockdale, T.: Understanding El Niño in ocean–atmosphere general circulation models: Progress and challenges, B. Am. Meteorol. Soc., 90, 325–340, https://doi.org/10.1175/2008BAMS2387.1, 2009.
Guilyardi, E., Capotondi, A., Lengaigne, M., Thual, S., and Wittenberg, A. T.: ENSO modelling: history, progress and challenges, in: El Niño in a changing climate, edited by: McPhaden, M. J., Santoso, A., Cai, W., AGU monograph, ISBN 9781119548164, https://doi.org/10.1002/9781119548164.ch9, 2020.
Gutowski Jr., W. J., Giorgi, F., Timbal, B., Frigon, A., Jacob, D., Kang, H.-S., Raghavan, K., Lee, B., Lennard, C., Nikulin, G., O'Rourke, E., Rixen, M., Solman, S., Stephenson, T., and Tangang, F.: WCRP COordinated Regional Downscaling EXperiment (CORDEX): a diagnostic MIP for CMIP6, Geosci. Model Dev., 9, 4087–4095, https://doi.org/10.5194/gmd-9-4087-2016, 2016.
Haarsma, R. J., Roberts, M. J., Vidale, P. L., Senior, C. A., Bellucci, A., Bao, Q., Chang, P., Corti, S., Fučkar, N. S., Guemas, V., von Hardenberg, J., Hazeleger, W., Kodama, C., Koenigk, T., Leung, L. R., Lu, J., Luo, J.-J., Mao, J., Mizielinski, M. S., Mizuta, R., Nobre, P., Satoh, M., Scoccimarro, E., Semmler, T., Small, J., and von Storch, J.-S.: High Resolution Model Intercomparison Project (HighResMIP v1.0) for CMIP6, Geosci. Model Dev., 9, 4185–4208, https://doi.org/10.5194/gmd-9-4185-2016, 2016.
Hannah, W. M., Bradley, A. M., Guba, O., Tang, Q., Golaz, J.-C., and Wolfe, J. D.: Separating physics and dynamics grids for improved computational efficiency in spectral element Earth system models, J. Adv. Model. Earth Sy., 13, e2020MS002419, https://doi.org/10.1029/2020ms002419, 2021.
Hassan, K. A., Rönnberg, N., Forsell, C., Cooper, M., and Johansson, J.: A study on 2D and 3D parallel coordinates for pattern identification in temporal multivariate data, in: 2019 23rd International Conference Information Visualisation (IV), 145–150, https://doi.org/10.1109/IV.2019.00033, 2019.
Hassell, D., Gregory, J., Blower, J., Lawrence, B. N., and Taylor, K. E.: A data model of the Climate and Forecast metadata conventions (CF-1.6) with a software implementation (cf-python v2.1), Geosci. Model Dev., 10, 4619–4646, https://doi.org/10.5194/gmd-10-4619-2017, 2017.
Hausfather, Z., Marvel, K., Schmidt, G. A., Nielsen-Gammon, J. W., and Zelinka, M.: Climate simulations: Recognize the “hot model” problem, Nature, 605, 26–29, https://doi.org/10.1038/d41586-022-01192-2, 2022.
Held, I. M., Guo, H., Adcroft, A., Dunne, J. P., Horowitz, L. W., Krasting, J., Shevliakova, E., Winton, M., Zhao, M., Bushuk, M., Wittenberg, A. T., and coauthors: Structure and performance of GFDL's CM4.0 climate model, J. Adv. Model. Earth Sy., 11, 3691–3727, https://doi.org/10.1029/2019MS001829, 2019.
Hendon, H. H., Zhang, C., and Glick, J. D.: Interannual Variation of the Madden–Julian Oscillation during Austral Summer, J. Climate, 12, 2538–2550, 1999.
Herger, N., Abramowitz, G., Knutti, R., Angélil, O., Lehmann, K., and Sanderson, B. M.: Selecting a climate model subset to optimise key ensemble properties, Earth Syst. Dynam., 9, 135–151, https://doi.org/10.5194/esd-9-135-2018, 2018.
Hersbach, H., Bell, B., Berrisford, P., Hirahara, S., Horányi, A., Muñoz-Sabater, J., Nicolas, J., Peubey, C., Radu, R., Schepers, D., and coauthors: The ERA5 global reanalysis, Q. J. Roy. Meteor. Soc., 146, 1999–2049, https://doi.org/10.1002/qj.3803, 2020.
Hintze, J. L. and Nelson, R. D.: Violin plots: A box plot-density trace synergism, Am. Stat., 52, 181–184, https://doi.org/10.1080/00031305.1998.10480559, 1998.
Hoyer, S. and Hamman, J.: xarray: N-D labeled Arrays and Datasets in Python, J. Open Res. Software, 5, 10, https://doi.org/10.5334/jors.148, 2017.
Huffman, G. J., Adler, R. F., Morrissey, M. M., Bolvin, D. T., Curtis, S., Joyce, R., McGavock, B., and Susskind, J.: Global precipitation at one-degree daily resolution from multisatellite observations, J. Hydrometeorol., 2, 36–50, 2001.
Huffman, G. J., Bolvin, D. T., Braithwaite, D., Hsu, K., Joyce, R., Kidd, C., Nelkin, E. J., Sorooshian, S., Tan, J., and Xie, P.: NASA global precipitation measurement (GPM) integrated multi-satellite retrievals for GPM (IMERG), Algorithm theoretical basis document (ATBD) version, 4, p. 30, 2015.
Inselberg, A.: Multidimensional detective, in: Proceedings of IEEE Symposium on Information Visualization, 100–107, https://doi.org/10.1109/INFVIS.1997.636793, 1997.
Inselberg, A.: Parallel Coordinates: Visualization, Exploration and Classification of High-Dimensional Data, in: Handbook of Data Visualization, edited by: Chen, C., Härdle, W., and Unwin, A., Springer, Berlin, Heidelberg, Germany, 643–680, https://doi.org/10.1007/978-3-540-33037-0_25, 2008.
Inselberg, A.: Parallel Coordinates, in: Encyclopedia of Database Systems, Springer, edited by: Liu, L., and Özsu, M. T., Springer, New York, NY, U.S.A., https://doi.org/10.1007/978-1-4899-7993-3_262-2, 2016.
Jakob, C., Gettelman, A., and Pitman, A.: The need to operationalize climate modelling, Nat. Clim. Change, 13, 1158–1160, https://doi.org/10.1038/s41558-023-01849-4, 2023.
Johansson, J. and Forsell, C.: Evaluation of parallel coordinates: Overview, categorization and guidelines for future research, IEEE T. Vis. Comput. G. R., 22, 579–588, https://doi.org/10.1109/TVCG.2015.2466992, 2016.
Joyce, R. J., Janowiak, J. E., Arkin, P. A., and Xie, P.: CMORPH: A method that produces global precipitation estimates from passive microwave and infrared data at high spatial and temporal resolution, J. Hydrometeorol., 5, 487–503, 2004.
Kang, D., Kim, D. H., Ahn, M.-S., Neale, R., Lee, J., and Gleckler, P. J.: The role of the mean state on MJO simulation in CESM2 ensemble simulation, Geophys. Res. Lett., 47, e2020GL089824, https://doi.org/10.1029/2020gl089824, 2020.
Kim, D., Sperber, K. R., Stern, W., Waliser, D. E., Kang, I. S., Maloney, E. D., Wang, W., Weickmann, K. M., Benedict, J. J., Khairoutdinov, M., Lee, M.-I., Neale, R., Suarez, M. J., Thayer-Calder, K., and Zhang, G.: Application of MJO simulation diagnostics to climate models, J. Climate, 22, 6413–6436, https://doi.org/10.1175/2009jcli3063.1, 2009.
Kim, H., Caron, J. M., Richter, J. H. and Simpson, I. R.: The lack of QBO-MJO connection in CMIP6 models, Geophys. Res. Lett., 47, e2020GL087295, https://doi.org/10.1029/2020GL087295, 2020.
Klein, S. A., Zhang, Y., Zelinka, M. D., Pincus, R., Boyle, J., and Gleckler, P. J.: Are climate model simulations of clouds improving? An evaluation using the ISCCP simulator, J. Geophys. Res.-Atmos., 118, 1329–1342, https://doi.org/10.1002/jgrd.50141, 2013.
Klingaman, N. P., Martin, G. M., and Moise, A.: ASoP (v1.0): a set of methods for analyzing scales of precipitation in general circulation models, Geosci. Model Dev., 10, 57–83, https://doi.org/10.5194/gmd-10-57-2017, 2017.
Knutti, R.: The end of model democracy? Climatic Change, 102, 395–404, https://doi.org/10.1007/s10584-010-9800-2, 2010.
Knutti, R., Abramowitz, G., Collins, M., Eyring, V., Gleckler, P. J., Hewitson, B., and Mearns, L.: Good Practice Guidance Paper on Assessing and Combining Multi Model Climate Projections, in: Meeting Report of the Intergovernmental Panel on Climate Change Expert Meeting on Assessing and Combining Multi Model Climate Projections, edited by: Stocker, T. F., Qin, D., Plattner, G.-K., Tignor, M., and Midgley, P. M., IPCC Working Group I Technical Support Unit, University of Bern, Bern, Switzerland, 2010.
Knutti, R., Sedláček, J., Sanderson, B. M., Lorenz, R., Fischer, E. M., and Eyring, V.: A climate model projection weighting scheme accounting for performance and interdependence, Geophys. Res. Lett., 44, 1909–1918, https://doi.org/10.1002/2016gl072012, 2017.
Labe, Z. M. and Barnes, E. A.: Comparison of Climate Model Large Ensembles With Observations in the Arctic Using Simple Neural Networks, Earth Space Sci., 9, e2022EA002348, https://doi.org/10.1029/2022EA002348, 2022.
Lambert, S. J. and Boer, G. J.: CMIP1 evaluation and intercomparison of coupled climate models, Clim. Dynam., 17, 83–106, https://doi.org/10.1007/PL00013736, 2001.
Lee, H., Goodman, A., McGibbney, L., Waliser, D. E., Kim, J., Loikith, P. C., Gibson, P. B., and Massoud, E. C.: Regional Climate Model Evaluation System powered by Apache Open Climate Workbench v1.3.0: an enabling tool for facilitating regional climate studies, Geosci. Model Dev., 11, 4435–4449, https://doi.org/10.5194/gmd-11-4435-2018, 2018.
Lee, J., Gleckler, P., Sperber, K., Doutriaux, C., and Williams, D.: High-dimensional Data Visualization for Climate Model Intercomparison: Application of the Circular Plot, in: Proceedings of the 8th International Workshop on Climate Informatics: CI 2018, NCAR Technical Note NCAR/TN-550+PROC, 12–14, https://doi.org/10.5065/D6BZ64XQ, 2018.
Lee, J., Sperber, K. R., Gleckler, P. J., Bonfils, C., and Taylor, K. E.: Quantifying the agreement between observed and simulated extratropical modes of interannual variability, Clim. Dynam., 52, 4057–4089, https://doi.org/10.1007/s00382-018-4355-4, 2019a.
Lee, J., Xue, Y., De Sales, F., Diallo, I., Marx, L., Ek, M., Sperber, K. R., and Gleckler, P. J.: Evaluation of multi-decadal UCLA-CFSv2 simulation and impact of interactive atmospheric-ocean feedback on global and regional variability, Clim. Dynam., 52, 3683–3707, https://doi.org/10.1007/s00382-018-4351-8, 2019b.
Lee, J., Planton, Y., Gleckler, P. J., Sperber, K. R., Guilyardi, E., Wittenberg, A. T., McPhaden, M. J., and Pallotta, G.: Robust evaluation of ENSO in climate models: How many ensemble members are needed?, Geophys. Res. Lett., 48, e2021GL095041, https://doi.org/10.1029/2021gl095041, 2021a.
Lee, J., Sperber, K. R., Gleckler, P. J., Taylor, K. E., and Bonfils, C.: Benchmarking performance changes in the simulation of extratropical modes of variability across CMIP generations, J. Climate, 34, 6945–6969, https://doi.org/10.1175/jcli-d-20-0832.1, 2021b.
Lee, J., Ahn, M.-S., Ordonez, A., Gleckler, P., and Ullrich, P.: PCMDI/pcmdi_metrics_results_archive, Zenodo [data set], https://doi.org/10.5281/zenodo.10181201, 2023a.
Lee, J., Gleckler, P., Ordonez, A., Ahn, M.-S., Ullrich, P., Tom, V., Jason, B., Charles, D., Durack, P., Shaheen, Z., Muryanto, L., Painter, J., and Krasting, J.: PCMDI/pcmdi_metrics: PMP Version 3.1.1, Zenodo [code], https://doi.org/10.5281/zenodo.592790, 2023b.
Leung, L. R., Boos, W. R., Catto, J. L., DeMott, C. A., Martin, G. M., Neelin, J. D., O'Brien, T. A., Xie, S., Feng, Z., Klingaman, N. P. Kuo, Y.-H., Lee, R. W., Martinez-Villalobos, C., Vishnu S., Priestley, M. D. K., Tao, C., and Zhou, Y.: Exploratory precipitation metrics: Spatiotemporal characteristics, process-oriented, and phenomena-based, J. Climate, 35, 3659–3686, https://doi.org/10.1175/JCLI-D-21-0590.1, 2022.
Lin, J.-P., Kiladis, G. N., Mapes, B. E., Weickmann, K. M., Sperber, K. R., Lin, W., Wheeler, M. C., Schubert, S. D., Del Genio, A. D., Donner, L. J., Emori, S., Guérémy, J.-F., Hourdin, F., Rasch, P. J., Roeckner, E., and Scinocca, J.: Tropical intraseasonal variability in 14 IPCC AR4 climate Models. Part I: Convective Signals, J. Climate, 19, 2665–2690, https://doi.org/10.1175/jcli3735.1, 2006.
Lin, Y., Huang, X., Liang, Y., Qin, Y., Xu, S., Huang, W., Xu, F., Liu, L., Wang, Y., Peng, Y., and Wang, L.: Community integrated earth system model (CIESM): Description and evaluation, J. Adv. Model. Earth Sy., 12, e2019MS002036, https://doi.org/10.1029/2019ms002036, 2020.
Loeb, N. G., Doelling, D. R., Wang, H., Su, W., Nguyen, C., Corbett, J. G., Liang, L., Mitrescu, C., Rose, F. G., and Seiji, K.: Clouds and the Earth's Radiant Energy System (CERES) Energy Balanced and Filled (EBAF) top-of-atmosphere (TOA) Edition-4.0 data product, Int. J. Climatol., 31, 895–918, https://doi.org/10.1175/JCLI-D-17-0208.1, 2018.
Longmate, J. M., Risser, M. D., and Feldman, D. R.: Prioritizing the selection of CMIP6 model ensemble members for downscaling projections of CONUS temperature and precipitation, Clim. Dynam., 61, 5171–5197, https://doi.org/10.1007/s00382-023-06846-z, 2023.
Lu, L., Wang, W. and Tan, Z.: Double-arc parallel coordinates and its axes re-ordering methods, Mobile Networks and Applications, 25, 1376–1391, https://doi.org/10.1007/s11036-019-01455-9, 2020.
Madden, R. A. and Julian, P.: Detection of a 40–50 day oscillation in the zonal wind in the Tropical Pacific, J. Atmos. Sci., 28, 702–708, https://doi.org/10.1175/1520-0469(1971)028, 1971.
Madden, R. A. and Julian, P.: Description of Global-Scale Circulation Cells in the Tropics with a 40–50 Day Period, J. Atmos. Sci., 29, 1109–1123, https://doi.org/10.1175/1520-0469(1972)029, 1972.
Madden, R. A. and Julian, P.: Observations of the 40–50-Day Tropical Oscillation – A Review, Mon. Weather Rev., 122, 814–837, https://doi.org/10.1175/1520-0493(1994)122, 1994.
Maloney, E. D., Gettelman, A., Ming, Y., Neelin, J. D., Barrie, D., Mariotti, A., Chen, C., Coleman, D., Kuo, Y. H., Singh, B., Annamalai, H., Berg, A., Booth, J. F., Camargo, S. J., Dai, A., Gonzalez, A., Hafner, J., Jiang, X., Jing, X., Kim, D. H., Kumar, A., Moon, Y., Naud, C. M., Sobel, A. H., Suzuki, K., Wang, F., Wang, J., Wing, A. A., Xu, X., and Zhao, M.: Process-Oriented evaluation of climate and weather forecasting models, B. Am. Meteorol. Soc., 100, 1665–1686, https://doi.org/10.1175/bams-d-18-0042.1, 2019.
Martin, G. M., Klingaman, N. P., and Moise, A. F.: Connecting spatial and temporal scales of tropical precipitation in observations and the MetUM-GA6, Geosci. Model Dev., 10, 105–126, https://doi.org/10.5194/gmd-10-105-2017, 2017.
McAvaney, B. J., Covey, C., Joussaume, S., Kattsov, V., Kitoh, A., Ogana, W., Pitman, A. J., Weaver, A. J., Wood, R. A., and Zhao, Z. C.: Model evaluation. In Climate Change 2001: The scientific basis, Contribution of WG1 to the Third Assessment Report of the IPCC (TAR) 471-523, Cambridge University Press, ISBN 0521 80767 0, 2001.
McPhaden, M. J., Zebiak, S. E., and Glantz, M. H.: ENSO as an integrating concept in Earth Science, Science, 314, 1740–1745, https://doi.org/10.1126/science.1132588, 2006.
McPhaden, M. J., Santoso, A., and Cai, W. (Eds.): El Niño Southern oscillation in a changing climate, American Geophysical Union, USA, 528 pp., ISBN 9781119548126, https://doi.org/10.1002/9781119548164, 2020.
Meehl, G. A., Boer, G. J., Covey, C., Latif, M., and Stouffer, R. J.: Intercomparison makes for a better climate model, Eos T. Am. Geophys. Un., 78, 445, https://doi.org/10.1029/97eo00276, 1997.
Meehl, G. A., Boer, G. J., Covey, C., Latif, M., and Stouffer, R. J.: The Coupled Model Intercomparison Project (CMIP), B. Am. Meteorol. Soc., 81, 313–318, 2000.
Meehl, G. A., Covey, C., Delworth, T. L., Latif, M., McAvaney, B. J., Mitchell, J. F. B., Stouffer, R. J., and Taylor, K. E.: THE WCRP CMIP3 Multimodel Dataset: A new era in climate change research, B. Am. Meteorol. Soc., 88, 1383–1394, https://doi.org/10.1175/bams-88-9-1383, 2007.
Merrifield, A. L., Brunner, L., Lorenz, R., Humphrey, V., and Knutti, R.: Climate model Selection by Independence, Performance, and Spread (ClimSIPS v1.0.1) for regional applications, Geosci. Model Dev., 16, 4715–4747, https://doi.org/10.5194/gmd-16-4715-2023, 2023.
Neelin, J. D., Krasting, J. P., Radhakrishnan, A., Liptak, J., Jackson, T. J., Ming, Y., Dong, W., Gettelman, A., Coleman, D., Maloney, E. D., Wing, A. A., Kuo, Y. H., Ahmed, F., Ullrich, P. A., Bitz, C. M., Neale, R., Ordonez, A., and Maroon, E.: Process-oriented diagnostics: principles, practice, community development and common standards, B. Am. Meteorol. Soc., 104, E1452–E1468, https://doi.org/10.1175/bams-d-21-0268.1, 2023.
Nowack, P., Runge, J., Eyring, V., and Haigh, J. D.: Causal networks for climate model evaluation and constrained projections, Nat. Commun., 11, 1415, https://doi.org/10.1038/s41467-020-15195-y, 2020.
Orbe, C., Van Roekel, L., Adames, Á. F., Dezfuli, A., Fasullo, J. T., Gleckler, P. J., Lee, J., Li, W., Nazarenko, L., Schmidt, G. A., Sperber, K. R., and Zhao, M.: Representation of modes of variability in six U.S. climate models, J. Climate, 33, 7591–7617, https://doi.org/10.1175/jcli-d-19-0956.1, 2020.
Ordonez, A. C., Klingaman, N. P., and Martin, G.: Analysing scales of precipitation, OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information), https://doi.org/10.11578/dc.20211029.5, 2021.
Papalexiou, S. M., Rajulapati, C. R., Clark, M. P., and Lehner, F.: Robustness of CMIP6 historical global mean temperature simulations: Trends, long-term persistence, autocorrelation, and distributional shape, Earth's Future, 8, e2020EF001667, https://doi.org/10.1029/2020EF001667, 2020.
Pascoe, C., Lawrence, B. N., Guilyardi, E., Juckes, M., and Taylor, K. E.: Documenting numerical experiments in support of the Coupled Model Intercomparison Project Phase 6 (CMIP6), Geosci. Model Dev., 13, 2149–2167, https://doi.org/10.5194/gmd-13-2149-2020, 2020.
PCMDI Simulation Summaries: https://pcmdi.llnl.gov/metrics/, last access: 8 May 2024.
Pendergrass, A. G., Gleckler, P. J., Leung, L. R., and Jakob, C.: Benchmarking simulated precipitation in earth system models, B. Am. Meteorol. Soc., 101, E814–E816, https://doi.org/10.1175/bams-d-19-0318.1, 2020.
Phillips, A. S., Deser, C., and Fasullo, J. T.: Evaluating modes of variability in climate models, Eos T. Am. Geophys. Un., 95, 453–455, https://doi.org/10.1002/2014eo490002, 2014.
Planton, Y., Guilyardi, E., Wittenberg, A. T., Lee, J., Gleckler, P. J., Bayr, T., McGregor, S., McPhaden, M. J., Power, S. B., Roehrig, R., Vialard, J., and Voldoire, A.: Evaluating Climate Models with the CLIVAR 2020 ENSO Metrics Package, B. Am. Meteorol. Soc., 102, E193–E217, https://doi.org/10.1175/bams-d-19-0337.1, 2021.
Planton, Y. Y., Lee, J., Wittenberg, A. T. Gleckler, P. J., Guilyardi, E., McGregor, S., and McPhaden, M. J.: Estimating uncertainty in simulated ENSO statistics, J. Adv. Model. Earth Sy., ESS Open Archive [preprint], https://doi.org10.22541/essoar.170196744.48068128/v1, 2023.
PMP Installation: http://pcmdi.github.io/pcmdi_metrics/install.html, last access: 8 May 2024.
Potter, G. L., Bader, D. C., Riches, M., Bamzai, A. and Joseph, R.: Celebrating two decades of the Program for Climate Model Diagnosis and Intercomparison, B. Am. Meteorol. Soc., 92, 629–631, https://doi.org/10.1175/2011BAMS3018.1, 2011.
Qin, Y., Zelinka, M. D., and Klein, S. A.: On the Correspondence Between Atmosphere-Only and Coupled Simulations for Radiative Feedbacks and Forcing From CO2, J. Geophys. Res.-Atmos., 127, e2021JD035460, https://doi.org/10.1029/2021jd035460, 2022.
Randall, D. A., Wood, R. A., Bony, S., Colman, R., Fichefet, T., Fyfe, J., Kattsov, V., Pitman, A., Shukla, J., Srinivasan, J., and Stouffer, R. J.: Climate models and their evaluation, in: Climate change 2007: The physical science basis. Contribution of Working Group I to the Fourth Assessment Report of the IPCC (FAR), Cambridge University Press, 589–662, ISBN 978-0-521-88009-1, 2007.
Rasch, P. J., Xie, S., Ma, P.-L., Lin, W., Wang, H., Tang, Q., Burrows, S. M., Caldwell, P., Zhang, K., Easter, R. C., Cameron-Smith, P. J., Singh, B., Wan, H., Golaz, J.-C., Harrop, B. E., Roesler, E. L., Bacmeister, J. T., Larson, V. E., Evans, K. J., Qian, Y., Taylor, M. A., Leung, L. R., Zhang, Y., Brent, L., Branstetter, M. L., Hannay, C., Mahajan, S., Mametjanov, A., Neale, R., Richter, J. H., Yoon, J.-H., Zender, C. S., Bader, D. C., Flanner, M., Foucar, J. G., Jacob, R., Keen, N. D., Klein, S. A., Liu, X., Salinger, A. G., Shrivastava, M., and Yang, Y.: An overview of the atmospheric component of the Energy Exascale Earth System model, J. Adv. Model. Earth Sy., 11, 2377–2411, https://doi.org/10.1029/2019ms001629, 2019.
Reed, K. A., Goldenson, N., Grotjahn, R., Gutowski, W. J., Jagannathan, K., Jones, A. D., Leung, L. R., McGinnis, S. A., Pryor, S. C., Srivastava, A. K., Ullrich, P. A., and Zarzycki, C. M.: Metrics as tools for bridging climate science and applications, WIREs Climate Change, 13, e799, https://doi.org/10.1002/wcc.799, 2022.
Reichler, T. and Kim, J.: How well do coupled models simulate today's climate?, B. Am. Meteorol. Soc., 89, 303–312, https://doi.org/10.1175/bams-89-3-303, 2008.
Righi, M., Andela, B., Eyring, V., Lauer, A., Predoi, V., Schlund, M., Vegas-Regidor, J., Bock, L., Brötz, B., de Mora, L., Diblen, F., Dreyer, L., Drost, N., Earnshaw, P., Hassler, B., Koldunov, N., Little, B., Loosveldt Tomas, S., and Zimmermann, K.: Earth System Model Evaluation Tool (ESMValTool) v2.0 – technical overview, Geosci. Model Dev., 13, 1179–1199, https://doi.org/10.5194/gmd-13-1179-2020, 2020.
Sanderson, B. M. and Wehner, M. F.: Weighting strategy for the Fourth National Climate Assessment, in: Climate Science Special Report: Fourth National Climate Assessment, Volume I, edited by: Wuebbles, D. J., Fahey, D. W., Hibbard, K. A., Dokken, D. J., Stewart, B. C., and Maycock, T. K., U.S. Global Change Research Program, Washington, DC, USA, 436–442, https://doi.org/10.7930/J06T0JS3, 2017.
Sanderson, B. M., Wehner, M., and Knutti, R.: Skill and independence weighting for multi-model assessments, Geosci. Model Dev., 10, 2379–2395, https://doi.org/10.5194/gmd-10-2379-2017, 2017.
Sherwood, S. C., Webb, M. J., Annan, J. D., Armour, K. C., Forster, P. M., Hargreaves, J. C., Hegerl, G. C., Klein, S. A., Marvel, K., Rohling, E. J., Watanabe, M., Andrews, T., Braconnot, P., Bretherton, C. S., Foster, G. L., Hausfather, Z., Von Der Heydt, A. S., Knutti, R., Mauritsen, T., Norris, J. R., Proistosescu, C., Rugenstein, M., Schmidt, G. A., Tokarska, K., and Zelinka, M. D.: An assessment of Earth's climate sensitivity using multiple lines of evidence, Rev. Geophys., 58, e2019RG000678, https://doi.org/10.1029/2019rg000678, 2020.
Singh, R., and AchutaRao, K.: Sensitivity of future climate change and uncertainty over India to performance-based model weighting, Climatic Change, 160, 385–406, https://doi.org/10.1007/s10584-019-02643-y, 2020.
Sillmann, J., Kharin, V. V., Zhang, X., Zwiers, F. W., and Bronaugh, D.: Climate extremes indices in the CMIP5 multimodel ensemble: Part 1. Model evaluation in the present climate, J. Geophys. Res.-Atmos., 118, 1716–1733, https://doi.org/10.1002/jgrd.50203, 2013.
Sperber, K. R.: Madden-Julian variability in NCAR CAM2.0 and CCSM2.0, Clim. Dynam., 23, 259–278, https://doi.org/10.1007/s00382-004-0447-4, 2004.
Sperber, K. R. and Annamalai, H.: The use of fractional accumulated precipitation for the evaluation of the annual cycle of monsoons, Clim. Dynam., 43, 3219–3244, https://doi.org/10.1007/s00382-014-2099-3, 2014.
Sperber, K. R., Annamalai, H., Kang, I.-S., Kitoh, A., Moise, A., Turner, A., Wang, B., and Zhou, T.: The Asian summer monsoon: an intercomparison of CMIP5 vs. CMIP3 simulation of the late 20th century, Clim. Dynam., 41, 2711–2744, https://doi.org/10.1007/s00382-012-1607-6, 2013.
Sperber, K. R., Gualdi, S., Legutke, S., and Gayler, V.: The Madden–Julian oscillation in ECHAM4 coupled and uncoupled general circulation models, Clim. Dynam., 25, 117–140, https://doi.org/10.1007/s00382-005-0026-3, 2005.
Srivastava, A., Grotjahn, R., and Ullrich, P. A.: Evaluation of historical CMIP6 model simulations of extreme precipitation over contiguous US regions, Weather Climate Extremes, 29, 100268, https://doi.org/10.1016/j.wace.2020.100268, 2020.
Steed, C. A., Shipman, G., Thornton, P., Ricciuto, D., Erickson, D. and Branstetter, M.: Practical application of parallel coordinates for climate model analysis, Procedia Comput. Sci., 9, 877-886, https://doi.org/10.1016/j.procs.2012.04.094, 2012.
Stevens, B., Satoh, M., Auger, L., Biercamp, J., Bretherton, C. S., Chen, X., Düben, P., Judt, F., Khairoutdinov, M., Klocke, D., Kodama, C., Kornblueh, L., Lin, S.-J., Neumann, P., Putman, W. M., Röber, N., Shibuya, R., Vanniere, B., Vidale, P. L., Wedi, N., and Zhou, L.: DYAMOND: the DYnamics of the Atmospheric general circulation Modeled On Non-hydrostatic Domains, Prog. Earth Planet. Sci., 6, 61, https://doi.org/10.1186/s40645-019-0304-z, 2019.
Stoner, A. M. K., Hayhoe, K., and Wuebbles, D. J.: Assessing general circulation model simulations of atmospheric teleconnection patterns, J. Climate, 22, 4348–4372, https://doi.org/10.1175/2009jcli2577.1, 2009.
Sung, H. M., Kim, J., Shim, S., Seo, J., Kwon, S.-H., Sun, M.-A., Moon, H.-J., Lee, J., Lim, Y. C., Boo, K.-O., Kim, Y., Lee, J., Lee, J., Kim, J.-S., Marzin, C., and Byun, Y.-H.: Climate change projection in the Twenty-First Century simulated by NIMS-KMA CMIP6 model based on new GHGs concentration pathways, Asia-Pac. J. Atmos. Sci., 57, 851–862, https://doi.org/10.1007/s13143-021-00225-6, 2021.
Tang, Q., Prather, M. J., Hsu, J., Ruiz, D. J., Cameron-Smith, P. J., Xie, S., and Golaz, J.-C.: Evaluation of the interactive stratospheric ozone (O3v2) module in the E3SM version 1 Earth system model, Geosci. Model Dev., 14, 1219–1236, https://doi.org/10.5194/gmd-14-1219-2021, 2021.
Tang, S., Fast, J. D., Zhang, K., Hardin, J. C., Varble, A. C., Shilling, J. E., Mei, F., Zawadowicz, M. A., and Ma, P.-L.: Earth System Model Aerosol–Cloud Diagnostics (ESMAC Diags) package, version 1: assessing E3SM aerosol predictions using aircraft, ship, and surface measurements, Geosci. Model Dev., 15, 4055–4076, https://doi.org/10.5194/gmd-15-4055-2022, 2022.
Tang, S., Varble, A. C., Fast, J. D., Zhang, K., Wu, P., Dong, X., Mei, F., Pekour, M., Hardin, J. C., and Ma, P.-L.: Earth System Model Aerosol–Cloud Diagnostics (ESMAC Diags) package, version 2: assessing aerosols, clouds, and aerosol–cloud interactions via field campaign and long-term observations, Geosci. Model Dev., 16, 6355–6376, https://doi.org/10.5194/gmd-16-6355-2023, 2023.
Taylor, K. E.: Summarizing multiple aspects of model performance in a single diagram, J. Geophys. Res., 106, 7183–7192, https://doi.org/10.1029/2000jd900719, 2001.
Taylor, K. E.: Truly conserving with conservative remapping methods, Geosci. Model Dev., 17, 415–430, https://doi.org/10.5194/gmd-17-415-2024, 2024.
Taylor, K. E., Stouffer, R. J., and Meehl, G. A.: An overview of CMIP5 and the experiment design, B. Am. Meteorol. Soc., 93, 485–498, https://doi.org/10.1175/bams-d-11-00094.1, 2012.
Teixeira, J., Waliser, D. E., Ferraro, R., Gleckler, P. J., Lee, T., and Potter, G. L.: Satellite observations for CMIP5: The Genesis of OBS4MIPs, B. Am. Meteorol. Soc., 95, 1329–1334, https://doi.org/10.1175/bams-d-12-00204.1, 2014.
Tian, B. and Dong, X.: The Double-ITCZ Bias in CMIP3, CMIP5, and CMIP6 Models Based on Annual Mean Precipitation, Geophys. Res. Lett., 47, e2020GL087232, https://doi.org/10.1029/2020GL087232, 2020.
Ullrich, P. A. and Zarzycki, C. M.: TempestExtremes: a framework for scale-insensitive pointwise feature tracking on unstructured grids, Geosci. Model Dev., 10, 1069–1090, https://doi.org/10.5194/gmd-10-1069-2017, 2017.
Ullrich, P. A., Zarzycki, C. M., McClenny, E. E., Pinheiro, M. C., Stansfield, A. M., and Reed, K. A.: TempestExtremes v2.1: a community framework for feature detection, tracking, and analysis in large datasets, Geosci. Model Dev., 14, 5023–5048, https://doi.org/10.5194/gmd-14-5023-2021, 2021.
U.S. Department of Energy (DOE): Benchmarking Simulated Precipitation in Earth System Models Workshop Report, DOE/SC-0203, U.S. Department of Energy Office of Science, Biological and Environmental Research (BER) Program. Germantown, Maryland, USA, 2020.
Vo, T., Po-Chedley, P., Boutte, J., Zhang, C., Lee, J., Gleckler, P., Durack, P., Taylor, K., and Golaz, J.-C.: Xarray Climate Data Analysis Tools (xCDAT): A Python Package for Simple and Robust Analysis of Climate Data, The 103rd AMS Annual Meeting, Abstract, 8–12 January, 2023, in Denver, Colorado, 11.3, 412648, 2023.
Waliser, D., Gleckler, P. J., Ferraro, R., Taylor, K. E., Ames, S., Biard, J., Bosilovich, M. G., Brown, O., Chepfer, H., Cinquini, L., Durack, P. J., Eyring, V., Mathieu, P.-P., Lee, T., Pinnock, S., Potter, G. L., Rixen, M., Saunders, R., Schulz, J., Thépaut, J.-N., and Tuma, M.: Observations for Model Intercomparison Project (Obs4MIPs): status for CMIP6, Geosci. Model Dev., 13, 2945–2958, https://doi.org/10.5194/gmd-13-2945-2020, 2020.
Waliser, D. E., Sperber, K. R., Hendon, H. H., Kim, D., Maloney, E. D., Wheeler, M. C., Weickmann, K. M., Zhang, C., Donner, L. J., Gottschalck, J., Higgins, W., Kang, I. S., Legler, D. M., Moncrieff, M. W., Schubert, S. D., Stern, W., Vitart, F., Wang, B., Wang, W., and Woolnough, S. J.: MJO Simulation Diagnostics, J. Climate, 22, 3006–3030, https://doi.org/10.1175/2008jcli2731.1, 2009.
Wang, J., Liu, X., Shen, H. W., and Lin, G.: Multi-resolution climate ensemble parameter analysis with nested parallel coordinates plots, IEEE T. Vis. Comput. G. R., 23, 81–90, https://doi.org/10.1109/TVCG.2016.2598830, 2017.
Wehner, M., Gleckler, P. J., and Lee, J.: Characterization of long period return values of extreme daily temperature and precipitation in the CMIP6 models: Part 1, model evaluation, Weather Climate Extremes, 30, 100283, https://doi.org/10.1016/j.wace.2020.100283, 2020.
Wehner, M., Lee, J., Risser, M. D., Ullrich, P. A., Gleckler, P. J., and Collins, W. D.: Evaluation of extreme sub-daily precipitation in high-resolution global climate model simulations, Philos. T. R. Soc. A., 379, 20190545, https://doi.org/10.1098/rsta.2019.0545, 2021.
Williams, D. N.: Visualization and analysis tools for ultrascale climate data, Eos T. Am. Geophys. Un., 95, 377–378, https://doi.org/10.1002/2014eo420002, 2014.
Williams, D. N., Doutriaux, C., Drach, R., and McCoy, R.: The Flexible Climate Data Analysis Tools (CDAT) for Multi-model Climate Simulation Data, IEEE International Conference on Data Mining Workshops, 254–261, https://doi.org/10.1109/icdmw.2009.64, 2009.
Williams, D. N., Balaji, V., Cinquini, L., Denvil, S., Duffy, D. Q., Evans, B., Ferraro, R., Hansen, R., Lautenschlager, M., and Trenham, C.: A global repository for Planet-Sized experiments and observations, B. Am. Meteorol. Soc., 97, 803–816, https://doi.org/10.1175/bams-d-15-00132.1, 2016.
Wong, P. C., Shen, H. W., Leung, R., Hagos, S., Lee, T. Y., Tong, X. and Lu, K.: Visual analytics of large-scale climate model data, in: 2014 IEEE 4th Symposium on Large Data Analysis and Visualization (LDAV), 85–92, https://doi.org/10.1109/LDAV.2014.7013208, 2014.
Xie, P., Joyce, R., Wu, S., Yoo, S. H., Yarosh, Y., Sun, F. and Lin, R.: Reprocessed, bias-corrected CMORPH global high-resolution precipitation estimates from 1998, J. Hydrometeorol., 18, 1617–1641, 2017.
Xue, Z. and Ullrich, P. A.: A Comprehensive Intermediate-Term Drought Evaluation System and Evaluation of Climate Data Products over the Conterminous United States, J. Hydrometeorol., 22, 2311–2337, https://doi.org/10.1175/jhm-d-20-0314.1, 2021.
Young, A. H., Knapp, K. R., Inamdar, A., Hankins, W., and Rossow, W. B.: The International Satellite Cloud Climatology Project H-Series climate data record product, Earth Syst. Sci. Data, 10, 583–593, https://doi.org/10.5194/essd-10-583-2018, 2018.
Zelinka, M. D., Myers, T. A., McCoy, D. T., Po-Chedley, S., Caldwell, P. M., Ceppi, P., Klein, S. A., and Taylor, K. E.: Causes of higher climate sensitivity in CMIP6 models, Geophys. Res. Lett., 47, e2019GL085782, https://doi.org/10.1029/2019GL085782, 2020.
Zelinka, M. D., Klein, S. A., Qin, Y., and Myers, T. A.: Evaluating climate models' cloud feedbacks against expert judgment, J. Geophys. Res.-Atmos., 127, e2021JD035198, https://doi.org/10.1029/2021jd035198, 2022.
Zhang, C. and Hendon, H. H.: Propagating and standing components of the intraseasonal oscillation in tropical convection, J. Atmos. Sci., 54, 741–752, https://doi.org/10.1175/1520-0469(1997)054, 1997.
Zhang, C., Xie, S., Klein, S. A., Ma, H. Y., Tang, S., Van Weverberg, K., Morcrette, C. J., and Petch, J.: CAUSES: Diagnosis of the summertime warm bias in CMIP5 climate models at the ARM Southern Great Plains site. J. Geophys. Res.-Atmos., 123, 2968–2992, https://doi.org/10.1002/2017JD027200, 2018.
Zhang, C., Xie, S., Tao, C., Tang, S., Emmenegger, T., Neelin, J. D., Schiro, K. A., Lin, W., and Shaheen, Z.: The ARM data-oriented metrics and diagnostics package for climate models: A new tool for evaluating climate models with field data, B. Am. Meteorol. Soc., 101, E1619-E1627, https://doi.org/10.1175/BAMS-D-19-0282.1, 2020.
Zhang, C., Golaz, J.-C., Forsyth, R., Vo, T., Xie, S., Shaheen, Z., Potter, G. L., Asay-Davis, X. S., Zender, C. S., Lin, W., Chen, C.-C., Terai, C. R., Mahajan, S., Zhou, T., Balaguru, K., Tang, Q., Tao, C., Zhang, Y., Emmenegger, T., Burrows, S., and Ullrich, P. A.: The E3SM Diagnostics Package (E3SM Diags v2.7): a Python-based diagnostics package for Earth system model evaluation, Geosci. Model Dev., 15, 9031–9056, https://doi.org/10.5194/gmd-15-9031-2022, 2022.
Zhao, B., Lin, P., Hu, A., Liu, H., Ding, M., Yu, Z., and Yu, Y.: Uncertainty in Atlantic Multidecadal Oscillation derived from different observed datasets and their possible causes, Front. Mar. Sci., 9, 1007646, https://doi.org/10.3389/fmars.2022.1007646, 2022.
Zhao, M., Golaz, J.-C., Held, I. M., Guo, H., Balaji, V., Benson, R., Chen, J. H., Chen, X., Donner, L. J., Dunne, J., Dunne, K. A., Durachta, J., Fan, S.-M., Freidenreich, S. M., Garner, S. T., Ginoux, P., Harris, L., Horowitz, L. W., Krasting, J. P., Langenhorst, A. R., Zhi, L., Lin, P., Lin, S. J., Malyshev, S., Mason, E., Milly, P. C. D., Ming, Y., Naik, V., Paulot, F., Paynter, D., Phillipps, P. J., Radhakrishnan, A., Ramaswamy, V., Robinson, T., Schwarzkopf, D., Seman, C. J., Shevliakova, E., Shen, Z., Shin, H. H., Silvers, L. G., Wilson, J. R., Winton, M., Wittenberg, A. T., Wyman, B., and Xiang, B.: The GFDL Global Atmosphere and Land Model AM4.0/LM4.0: 1. Simulation characteristics with prescribed SSTS, J. Adv. Model. Earth Sy., 10, 691–734, https://doi.org/10.1002/2017ms001208, 2018.
Short summary
We introduce an open-source software, the PCMDI Metrics Package (PMP), developed for a comprehensive comparison of Earth system models (ESMs) with real-world observations. Using diverse metrics evaluating climatology, variability, and extremes simulated in thousands of simulations from the Coupled Model Intercomparison Project (CMIP), PMP aids in benchmarking model improvements across generations. PMP also enables efficient tracking of performance evolutions during ESM developments.
We introduce an open-source software, the PCMDI Metrics Package (PMP), developed for a...
