Articles | Volume 19, issue 13
https://doi.org/10.5194/gmd-19-6189-2026
© Author(s) 2026. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/gmd-19-6189-2026
© Author(s) 2026. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Model evaluation paper
| 
10 Jul 2026
Model evaluation paper |
| 10 Jul 2026
| 10 Jul 2026
New framework for benchmarking decadal predictions leveraging the PCMDI Metric Package with interactive visualization
Jung Choi
Yonsei University, Seoul, 03722, South Korea
Lawrence Livermore National Laboratory, Livermore, 94550, United States
Kristin Chang
Lawrence Livermore National Laboratory, Livermore, 94550, United States
Paul A. Ullrich
Lawrence Livermore National Laboratory, Livermore, 94550, United States
UC Davis, Davis, 95616, United States
Peter J. Gleckler
Lawrence Livermore National Laboratory, Livermore, 94550, United States
Sang-Yoon Jun
Korea Polar Research Institute, Incheon, 21990, Korea
Related authors
No articles found.
Hyeong-Gyu Kim, Joowan Kim, Sang-Yoon Jun, Seong-Joong Kim, and Damwon So
EGUsphere, https://doi.org/10.5194/egusphere-2026-3834, https://doi.org/10.5194/egusphere-2026-3834, 2026
This preprint is open for discussion and under review for Climate of the Past (CP).
Short summary
Short summary
The westerly winds over the Southern Ocean influence ocean currents and the global carbon cycle, yet how they behaved during the last ice age remains uncertain. Using a global climate model, we tested how the expansion of Antarctic sea ice affected these winds. The sea ice cooled the surface, strengthening the winds and shifting them toward Antarctica, outweighing the opposite pull from a colder tropics. Our results show Antarctic sea ice is a key driver of these winds in past climates.
Giuliana Pallotta, Shiheng Duan, Celine Bonfils, Jiwoo Lee, Seth Goodnight, and Paul Ullrich
EGUsphere, https://doi.org/10.48550/arXiv.2604.06567, https://doi.org/10.48550/arXiv.2604.06567, 2026
This preprint is open for discussion and under review for Geoscientific Model Development (GMD).
Short summary
Short summary
Deep-learning Earth system models (DL-ESMs) offer a computationally efficient alternative to traditional Earth system models, but their suitability for Earth system applications requires rigorous evaluation. We apply the PCMDI Metrics Package to assess ACE2 and NeuralGCM using climatology, variability, monsoon, and precipitation diagnostics, providing one of the first systematic Earth system model evaluations of DL-ESMs.
Jishi Zhang, Jean-Christophe Golaz, Matthew Vincent Signorotti, Hsiang-He Lee, Peter Bogenschutz, Minda Monteagudo, Paul Aaron Ullrich, Robert S. Arthur, Stephen Po-Chedley, Philip Cameron-Smith, and Jean-Paul Watson
Geosci. Model Dev., 19, 4513–4545, https://doi.org/10.5194/gmd-19-4513-2026, https://doi.org/10.5194/gmd-19-4513-2026, 2026
Short summary
Short summary
We ran a convection-permitting model with regional mesh refinement (3.25 km and 800 m) to simulate present-day wind and solar capacity factors over California, coupling it to an energy generation model. The high-resolution models captured realistic seasonal and diurnal cycles, with wind markedly better than a 25 km model and solar outperforming a 3 km operational forecast. We highlight the critical role of resolution, modeling assumptions, and data reliability in renewable energy assessment.
Hongyu Chen, Paul A. Ullrich, Julian Panetta, David Marsico, Moritz Hanke, Rajeev Jain, Chengzhu Zhang, and Robert L. Jacob
EGUsphere, https://doi.org/10.5194/egusphere-2026-636, https://doi.org/10.5194/egusphere-2026-636, 2026
Short summary
Short summary
Accurate data transfer between different grids is essential in climate and weather modeling. This study analyzes the geometric operations that underpin such transfers on the spherical Earth, identifies gaps and limitations in current methods, and introduces improved algorithms to ensure accuracy, reliability, and performance for next-generation modeling and data analysis systems.
Forrest M. Hoffman, Birgit Hassler, Ranjini Swaminathan, Jared Lewis, Bouwe Andela, Nathaniel Collier, Dóra Hegedűs, Jiwoo Lee, Charlotte Pascoe, Mika Pflüger, Martina Stockhause, Paul Ullrich, Min Xu, Lisa Bock, Felicity Chun, Bettina K. Gier, Douglas I. Kelley, Axel Lauer, Julien Lenhardt, Manuel Schlund, Mohanan G. Sreeush, Katja Weigel, Ed Blockley, Rebecca Beadling, Romain Beucher, Demiso D. Dugassa, Valerio Lembo, Jianhua Lu, Swen Brands, Jerry Tjiputra, Elizaveta Malinina, Brian Mederios, Enrico Scoccimarro, Jeremy Walton, Philip Kershaw, André L. Marquez, Malcolm J. Roberts, Eleanor O’Rourke, Elisabeth Dingley, Briony Turner, Helene Hewitt, and John P. Dunne
EGUsphere, https://doi.org/10.5194/egusphere-2025-2685, https://doi.org/10.5194/egusphere-2025-2685, 2025
Short summary
Short summary
As Earth system models become more complex, rapid and comprehensive evaluation through comparison with observational data is necessary. The upcoming Assessment Fast Track for the Seventh Phase of the Coupled Model Intercomparison Project (CMIP7) will require fast analysis. This paper describes a new Rapid Evaluation Framework (REF) that was developed for the Assessment Fast Track that will be run at the Earth System Grid Federation (ESGF) to inform the community about the performance of models.
Katherine M. Smith, Alice M. Barthel, LeAnn M. Conlon, Luke P. Van Roekel, Anthony Bartoletti, Jean-Christophe Golaz, Chengzhu Zhang, Carolyn Branecky Begeman, James J. Benedict, Gautam Bisht, Yan Feng, Walter Hannah, Bryce E. Harrop, Nicole Jeffery, Wuyin Lin, Po-Lun Ma, Mathew E. Maltrud, Mark R. Petersen, Balwinder Singh, Qi Tang, Teklu Tesfa, Jonathan D. Wolfe, Shaocheng Xie, Xue Zheng, Karthik Balaguru, Oluwayemi Garuba, Peter Gleckler, Aixue Hu, Jiwoo Lee, Ben Moore-Maley, and Ana C. Ordoñez
Geosci. Model Dev., 18, 1613–1633, https://doi.org/10.5194/gmd-18-1613-2025, https://doi.org/10.5194/gmd-18-1613-2025, 2025
Short summary
Short summary
Version 2.1 of the U.S. Department of Energy's Energy Exascale Earth System Model (E3SM) adds the Fox-Kemper et al. (2011) mixed-layer eddy parameterization, which restratifies the ocean surface layer through an overturning streamfunction. Results include surface layer bias reduction in temperature, salinity, and sea ice extent in the North Atlantic; a small strengthening of the Atlantic meridional overturning circulation; and improvements to many atmospheric climatological variables.
Malcolm J. Roberts, Kevin A. Reed, Qing Bao, Joseph J. Barsugli, Suzana J. Camargo, Louis-Philippe Caron, Ping Chang, Cheng-Ta Chen, Hannah M. Christensen, Gokhan Danabasoglu, Ivy Frenger, Neven S. Fučkar, Shabeh ul Hasson, Helene T. Hewitt, Huanping Huang, Daehyun Kim, Chihiro Kodama, Michael Lai, Lai-Yung Ruby Leung, Ryo Mizuta, Paulo Nobre, Pablo Ortega, Dominique Paquin, Christopher D. Roberts, Enrico Scoccimarro, Jon Seddon, Anne Marie Treguier, Chia-Ying Tu, Paul A. Ullrich, Pier Luigi Vidale, Michael F. Wehner, Colin M. Zarzycki, Bosong Zhang, Wei Zhang, and Ming Zhao
Geosci. Model Dev., 18, 1307–1332, https://doi.org/10.5194/gmd-18-1307-2025, https://doi.org/10.5194/gmd-18-1307-2025, 2025
Short summary
Short summary
HighResMIP2 is a model intercomparison project focusing on high-resolution global climate models, that is, those with grid spacings of 25 km or less in the atmosphere and ocean, using simulations of decades to a century in length. We are proposing an update of our simulation protocol to make the models more applicable to key questions for climate variability and hazard in present-day and future projections and to build links with other communities to provide more robust climate information.
Bo Dong, Paul Ullrich, Jiwoo Lee, Peter Gleckler, Kristin Chang, and Travis A. O'Brien
Geosci. Model Dev., 18, 961–976, https://doi.org/10.5194/gmd-18-961-2025, https://doi.org/10.5194/gmd-18-961-2025, 2025
Short summary
Short summary
A metrics package designed for easy analysis of atmospheric river (AR) characteristics and statistics is presented. The tool is efficient for diagnosing systematic AR bias in climate models and useful for evaluating new AR characteristics in model simulations. In climate models, landfalling AR precipitation shows dry biases globally, and AR tracks are farther poleward (equatorward) in the North and South Atlantic (South Pacific and Indian Ocean).
Paul J. Durack, Karl E. Taylor, Peter J. Gleckler, Gerald A. Meehl, Bryan N. Lawrence, Curt Covey, Ronald J. Stouffer, Guillaume Levavasseur, Atef Ben-Nasser, Sebastien Denvil, Martina Stockhause, Jonathan M. Gregory, Martin Juckes, Sasha K. Ames, Fabrizio Antonio, David C. Bader, John P. Dunne, Daniel Ellis, Veronika Eyring, Sandro L. Fiore, Sylvie Joussaume, Philip Kershaw, Jean-Francois Lamarque, Michael Lautenschlager, Jiwoo Lee, Chris F. Mauzey, Matthew Mizielinski, Paola Nassisi, Alessandra Nuzzo, Eleanor O’Rourke, Jeffrey Painter, Gerald L. Potter, Sven Rodriguez, and Dean N. Williams
EGUsphere, https://doi.org/10.5194/egusphere-2024-3729, https://doi.org/10.5194/egusphere-2024-3729, 2025
Short summary
Short summary
CMIP6 was the most expansive and ambitious Model Intercomparison Project (MIP), the latest in a history, extending four decades. CMIP engaged a growing community focused on improving climate understanding, and quantifying and attributing observed climate change being experienced today. The project's profound impact is due to the combining the latest climate science and technology, enabling the latest-generation climate simulations and increasing community attention in every successive phase.
Seung H. Baek, Paul A. Ullrich, Bo Dong, and Jiwoo Lee
Geosci. Model Dev., 17, 8665–8681, https://doi.org/10.5194/gmd-17-8665-2024, https://doi.org/10.5194/gmd-17-8665-2024, 2024
Short summary
Short summary
We evaluate downscaled products by examining locally relevant co-variances during precipitation events. Common statistical downscaling techniques preserve expected co-variances during convective precipitation (a stationary phenomenon). However, they dampen future intensification of frontal precipitation (a non-stationary phenomenon) captured in global climate models and dynamical downscaling. Our study quantifies a ramification of the stationarity assumption underlying statistical downscaling.
Jiwoo Lee, Peter J. Gleckler, Min-Seop Ahn, Ana Ordonez, Paul A. Ullrich, Kenneth R. Sperber, Karl E. Taylor, Yann Y. Planton, Eric Guilyardi, Paul Durack, Celine Bonfils, Mark D. Zelinka, Li-Wei Chao, Bo Dong, Charles Doutriaux, Chengzhu Zhang, Tom Vo, Jason Boutte, Michael F. Wehner, Angeline G. Pendergrass, Daehyun Kim, Zeyu Xue, Andrew T. Wittenberg, and John Krasting
Geosci. Model Dev., 17, 3919–3948, https://doi.org/10.5194/gmd-17-3919-2024, https://doi.org/10.5194/gmd-17-3919-2024, 2024
Short summary
Short summary
We introduce an open-source software, the PCMDI Metrics Package (PMP), developed for a comprehensive comparison of Earth system models (ESMs) with real-world observations. Using diverse metrics evaluating climatology, variability, and extremes simulated in thousands of simulations from the Coupled Model Intercomparison Project (CMIP), PMP aids in benchmarking model improvements across generations. PMP also enables efficient tracking of performance evolutions during ESM developments.
Justin L. Willson, Kevin A. Reed, Christiane Jablonowski, James Kent, Peter H. Lauritzen, Ramachandran Nair, Mark A. Taylor, Paul A. Ullrich, Colin M. Zarzycki, David M. Hall, Don Dazlich, Ross Heikes, Celal Konor, David Randall, Thomas Dubos, Yann Meurdesoif, Xi Chen, Lucas Harris, Christian Kühnlein, Vivian Lee, Abdessamad Qaddouri, Claude Girard, Marco Giorgetta, Daniel Reinert, Hiroaki Miura, Tomoki Ohno, and Ryuji Yoshida
Geosci. Model Dev., 17, 2493–2507, https://doi.org/10.5194/gmd-17-2493-2024, https://doi.org/10.5194/gmd-17-2493-2024, 2024
Short summary
Short summary
Accurate simulation of tropical cyclones (TCs) is essential to understanding their behavior in a changing climate. One way this is accomplished is through model intercomparison projects, where results from multiple climate models are analyzed to provide benchmark solutions for the wider climate modeling community. This study describes and analyzes the previously developed TC test case for nine climate models in an intercomparison project, providing solutions that aid in model development.
Lele Shu, Paul Ullrich, Xianhong Meng, Christopher Duffy, Hao Chen, and Zhaoguo Li
Geosci. Model Dev., 17, 497–527, https://doi.org/10.5194/gmd-17-497-2024, https://doi.org/10.5194/gmd-17-497-2024, 2024
Short summary
Short summary
Our team developed rSHUD v2.0, a toolkit that simplifies the use of the SHUD, a model simulating water movement in the environment. We demonstrated its effectiveness in two watersheds, one in the USA and one in China. The toolkit also facilitated the creation of the Global Hydrological Data Cloud, a platform for automatic data processing and model deployment, marking a significant advancement in hydrological research.
Qi Tang, Jean-Christophe Golaz, Luke P. Van Roekel, Mark A. Taylor, Wuyin Lin, Benjamin R. Hillman, Paul A. Ullrich, Andrew M. Bradley, Oksana Guba, Jonathan D. Wolfe, Tian Zhou, Kai Zhang, Xue Zheng, Yunyan Zhang, Meng Zhang, Mingxuan Wu, Hailong Wang, Cheng Tao, Balwinder Singh, Alan M. Rhoades, Yi Qin, Hong-Yi Li, Yan Feng, Yuying Zhang, Chengzhu Zhang, Charles S. Zender, Shaocheng Xie, Erika L. Roesler, Andrew F. Roberts, Azamat Mametjanov, Mathew E. Maltrud, Noel D. Keen, Robert L. Jacob, Christiane Jablonowski, Owen K. Hughes, Ryan M. Forsyth, Alan V. Di Vittorio, Peter M. Caldwell, Gautam Bisht, Renata B. McCoy, L. Ruby Leung, and David C. Bader
Geosci. Model Dev., 16, 3953–3995, https://doi.org/10.5194/gmd-16-3953-2023, https://doi.org/10.5194/gmd-16-3953-2023, 2023
Short summary
Short summary
High-resolution simulations are superior to low-resolution ones in capturing regional climate changes and climate extremes. However, uniformly reducing the grid size of a global Earth system model is too computationally expensive. We provide an overview of the fully coupled regionally refined model (RRM) of E3SMv2 and document a first-of-its-kind set of climate production simulations using RRM at an economic cost. The key to this success is our innovative hybrid time step method.
Min-Seop Ahn, Paul A. Ullrich, Peter J. Gleckler, Jiwoo Lee, Ana C. Ordonez, and Angeline G. Pendergrass
Geosci. Model Dev., 16, 3927–3951, https://doi.org/10.5194/gmd-16-3927-2023, https://doi.org/10.5194/gmd-16-3927-2023, 2023
Short summary
Short summary
We introduce a framework for regional-scale evaluation of simulated precipitation distributions with 62 climate reference regions and 10 metrics and apply it to evaluate CMIP5 and CMIP6 models against multiple satellite-based precipitation products. The common model biases identified in this study are mainly associated with the overestimated light precipitation and underestimated heavy precipitation. These biases persist from earlier-generation models and have been slightly improved in CMIP6.
Abhishekh Kumar Srivastava, Paul Aaron Ullrich, Deeksha Rastogi, Pouya Vahmani, Andrew Jones, and Richard Grotjahn
Geosci. Model Dev., 16, 3699–3722, https://doi.org/10.5194/gmd-16-3699-2023, https://doi.org/10.5194/gmd-16-3699-2023, 2023
Short summary
Short summary
Stakeholders need high-resolution regional climate data for applications such as assessing water availability and mountain snowpack. This study examines 3 h and 24 h historical precipitation over the contiguous United States in the 12 km WRF version 4.2.1-based dynamical downscaling of the ERA5 reanalysis. WRF improves precipitation characteristics such as the annual cycle and distribution of the precipitation maxima, but it also displays regionally and seasonally varying precipitation biases.
Zeyu Xue, Paul Ullrich, and Lai-Yung Ruby Leung
Hydrol. Earth Syst. Sci., 27, 1909–1927, https://doi.org/10.5194/hess-27-1909-2023, https://doi.org/10.5194/hess-27-1909-2023, 2023
Short summary
Short summary
We examine the sensitivity and robustness of conclusions drawn from the PGW method over the NEUS by conducting multiple PGW experiments and varying the perturbation spatial scales and choice of perturbed meteorological variables to provide a guideline for this increasingly popular regional modeling method. Overall, we recommend PGW experiments be performed with perturbations to temperature or the combination of temperature and wind at the gridpoint scale, depending on the research question.
David H. Marsico and Paul A. Ullrich
Geosci. Model Dev., 16, 1537–1551, https://doi.org/10.5194/gmd-16-1537-2023, https://doi.org/10.5194/gmd-16-1537-2023, 2023
Short summary
Short summary
Climate models involve several different components, such as the atmosphere, ocean, and land models. Information needs to be exchanged, or remapped, between these models, and devising algorithms for performing this exchange is important for ensuring the accuracy of climate simulations. In this paper, we examine the efficacy of several traditional and novel approaches to remapping on the sphere and demonstrate where our approaches offer improvement.
Chengzhu Zhang, Jean-Christophe Golaz, Ryan Forsyth, Tom Vo, Shaocheng Xie, Zeshawn Shaheen, Gerald L. Potter, Xylar S. Asay-Davis, Charles S. Zender, Wuyin Lin, Chih-Chieh Chen, Chris R. Terai, Salil Mahajan, Tian Zhou, Karthik Balaguru, Qi Tang, Cheng Tao, Yuying Zhang, Todd Emmenegger, Susannah Burrows, and Paul A. Ullrich
Geosci. Model Dev., 15, 9031–9056, https://doi.org/10.5194/gmd-15-9031-2022, https://doi.org/10.5194/gmd-15-9031-2022, 2022
Short summary
Short summary
Earth system model (ESM) developers run automated analysis tools on data from candidate models to inform model development. This paper introduces a new Python package, E3SM Diags, that has been developed to support ESM development and use routinely in the development of DOE's Energy Exascale Earth System Model. This tool covers a set of essential diagnostics to evaluate the mean physical climate from simulations, as well as several process-oriented and phenomenon-based evaluation diagnostics.
Vijay S. Mahadevan, Jorge E. Guerra, Xiangmin Jiao, Paul Kuberry, Yipeng Li, Paul Ullrich, David Marsico, Robert Jacob, Pavel Bochev, and Philip Jones
Geosci. Model Dev., 15, 6601–6635, https://doi.org/10.5194/gmd-15-6601-2022, https://doi.org/10.5194/gmd-15-6601-2022, 2022
Short summary
Short summary
Coupled Earth system models require transfer of field data between multiple components with varying spatial resolutions to determine the correct climate behavior. We present the Metrics for Intercomparison of Remapping Algorithms (MIRA) protocol to evaluate the accuracy, conservation properties, monotonicity, and local feature preservation of four different remapper algorithms for various unstructured mesh problems of interest. Future extensions to more practical use cases are also discussed.
Paul A. Ullrich, Colin M. Zarzycki, Elizabeth E. McClenny, Marielle C. Pinheiro, Alyssa M. Stansfield, and Kevin A. Reed
Geosci. Model Dev., 14, 5023–5048, https://doi.org/10.5194/gmd-14-5023-2021, https://doi.org/10.5194/gmd-14-5023-2021, 2021
Short summary
Short summary
TempestExtremes (TE) is a multifaceted framework for feature detection, tracking, and scientific analysis of regional or global Earth system datasets. Version 2.1 of TE now provides extensive support for nodal and areal features. This paper describes the algorithms that have been added to the TE framework since version 1.0 and gives several examples of how these can be combined to produce composite algorithms for evaluating and understanding atmospheric features.
Cited articles
Adler, R. F., Gu, G., Sapiano, M., Wang, J.-J., and Huffman, G. J.: Global Precipitation: Means, Variations and Trends During the Satellite Era (1979–2014), Surv. Geophys., 38, 679–699, https://doi.org/10.1007/s10712-017-9416-4, 2017.
Bilbao, R., Wild, S., Ortega, P., Acosta-Navarro, J., Arsouze, T., Bretonnière, P.-A., Caron, L.-P., Castrillo, M., Cruz-García, R., Cvijanovic, I., Doblas-Reyes, F. J., Donat, M., Dutra, E., Echevarría, P., Ho, A.-C., Loosveldt-Tomas, S., Moreno-Chamarro, E., Pérez-Zanon, N., Ramos, A., Ruprich-Robert, Y., Sicardi, V., Tourigny, E., and Vegas-Regidor, J.: Assessment of a full-field initialized decadal climate prediction system with the CMIP6 version of EC-Earth, Earth Syst. Dynam., 12, 173–196, https://doi.org/10.5194/esd-12-173-2021, 2021.
Boer, G. J., Smith, D. M., Cassou, C., Doblas-Reyes, F., Danabasoglu, G., Kirtman, B., Kushnir, Y., Kimoto, M., Meehl, G. A., Msadek, R., Mueller, W. A., Taylor, K. E., Zwiers, F., Rixen, M., Ruprich-Robert, Y., and Eade, R.: The Decadal Climate Prediction Project (DCPP) contribution to CMIP6, Geosci. Model Dev., 9, 3751–3777, https://doi.org/10.5194/gmd-9-3751-2016, 2016.
Buontempo, C., Burgess, S. N, Dee, D., Pinty, B., Thépaut, J.-N., Rixen, M., Almond, S., Armstrong, D., Brookshaw, A., Alos, A. L., Bell, B., Bergeron, C., Cagnazzo, C., Comyn-Platt, E., Damasio-Da-Costa, E., Guillory, A., Hersbach, H., Horányi, A., Nicolas, J., Obregon, A., Ramos, E. P., Raoult, B., Muñoz-Sabater, J., Simmons, A., Soci, C., Suttie, M., Vamborg, F., Varndell, J., Vermoote, S., Yang, X., and de Marcillaand, J. G.: The Copernicus Climate Change Service: Climate Science in Action, B. Am. Meteorol. Soc., 103, E2669–E2687, https://doi.org/10.1175/BAMS-D-21-0315.1, 2022.
Choi, J. and Lee, J.: Dataset for PCMDI Metrics Package DCPP workflow and Metrics, Zenodo [data set], https://doi.org/10.5281/zenodo.20822040, 2026.
Choi, J. and Son, S.-W.: Seasonal-to-decadal prediction of El Niño–Southern Oscillation and Pacific Decadal Oscillation, NPJ Clim. Atmos. Sci., 5, 29, https://doi.org/10.1038/s41612-022-00251-9, 2022.
Choi, J., Son, S.-W., Ham, Y.-G., Lee, J., and Kim, H.: Seasonal-to-Interannual Prediction Skills of Near-Surface Air Temperature in the CMIP5 Decadal Hindcast Experiments, J. Climate, 29, 1511–1527, https://doi.org/10.1175/JCLI-D-15-0182.1, 2016.
Choi, J., Jun, S.-Y., Son, S.-W., Hyun, Y.-K., Lee, J.-R., Lee, J., Boo, K.-O., and Park, B.-J.: Near-term Climate Prediction of Agricultural Thermal Conditions in East Asia, Adv. Atmos. Sci., 43, 631–644, https://doi.org/10.1007/s00376-025-4471-0, 2026.
Chylek, P., Folland, C. K., Klett, J. D., Wang, M., Lesins, G., and Dubey, M. K.: High values of the Arctic Amplification in the early decades of the 21st century: Causes of discrepancy by CMIP6 models between observation and simulation, J. Geophys. Res.-Atmos., 128, e2023JD039269, https://doi.org/10.1029/2023JD039269, 2023.
Corti, S., Palmer, T., Balmaseda, M., Weisheimer, A., Drijfhout, S., Dunstone, N., Hazeleger, W., Kröger, J., Pohlmann, H., Smith, D.. von Storch, J.-S., and Wouters, B.: Impact of Initial Conditions versus External Forcing in Decadal Climate Predictions: A Sensitivity Experiment, J. Climate, 28, 4454–4470, https://doi.org/10.1175/JCLI-D-14-00671.1, 2015.
Dalaiden, Q., Goosse, H., Rezsöhazy, J., and Thomas, E. R.: Reconstructing atmospheric circulation and sea-ice extent in the West Antarctic over the past 200 years using data assimilation, Clim. Dynam., 57, 3479–3503, https://doi.org/10.1007/s00382-021-05879-6, 2021.
Dunne, J. P., Hewitt, H. T., Arblaster, J. M., Bonou, F., Boucher, O., Cavazos, T., Dingley, B., Durack, P. J., Hassler, B., Juckes, M., Miyakawa, T., Mizielinski, M., Naik, V., Nicholls, Z., O'Rourke, E., Pincus, R., Sanderson, B. M., Simpson, I. R., and Taylor, K. E.: An evolving Coupled Model Intercomparison Project phase 7 (CMIP7) and Fast Track in support of future climate assessment, Geosci. Model Dev., 18, 6671–6700, https://doi.org/10.5194/gmd-18-6671-2025, 2025.
Dunstone, N., Lockwood, J., Solaraju-Murali, B., Reinhardt, K., Tsartsali, E. E., Athanasiadis, P. J., Bellucci, A., Brookshaw, A., Caron, L.-P., Doblas-Reyes, F. J., Früh, B., González-Reviriego, N., Gualdi, S., Hermanson, L., Materia, S., Nicodemou, A., Nicolì, D., Pankatz, K., Paxian, A., Scaife, A., Smith, D., and Thornton, H. E.: Towards Useful Decadal Climate Services, B. Am. Meteorol. Soc., 103, E1705–E1719, https://doi.org/10.1175/BAMS-D-21-0190.1, 2022.
Eade, R., Smith, D., Scaife, A., Wallace, E., Dunstone, N., Hermanson, L., and Robinson, N.: Do seasonal-to-decadal climate predictions underestimate the predictability of the real world?, Geopys. Res. Lett., 41, 5620–5628, https://doi.org/10.1002/2014GL061146, 2014.
Eyring, V., Bock, L., Lauer, A., Righi, M., Schlund, M., Andela, B., Arnone, E., Bellprat, O., Brötz, B., Caron, L.-P., Carvalhais, N., Cionni, I., Cortesi, N., Crezee, B., Davin, E. L., Davini, P., Debeire, K., de Mora, L., Deser, C., Docquier, D., Earnshaw, P., Ehbrecht, C., Gier, B. K., Gonzalez-Reviriego, N., Goodman, P., Hagemann, S., Hardiman, S., Hassler, B., Hunter, A., Kadow, C., Kindermann, S., Koirala, S., Koldunov, N., Lejeune, Q., Lembo, V., Lovato, T., Lucarini, V., Massonnet, F., Müller, B., Pandde, A., Pérez-Zanón, N., Phillips, A., Predoi, V., Russell, J., Sellar, A., Serva, F., Stacke, T., Swaminathan, R., Torralba, V., Vegas-Regidor, J., von Hardenberg, J., Weigel, K., and Zimmermann, K.: Earth System Model Evaluation Tool (ESMValTool) v2.0 – an extended set of large-scale diagnostics for quasi-operational and comprehensive evaluation of Earth system models in CMIP, Geosci. Model Dev., 13, 3383–3438, https://doi.org/10.5194/gmd-13-3383-2020, 2020.
Gleckler, P. J., Taylor, K. E., and Doutriaux, C.: Performance metrics for climate models, J. Geophys. Res., 113, D06104, https://doi.org/10.1029/2007JD008972, 2008.
Goddard, L., Kumar, A., Solomon, A., Smith, D., Boer, G., Gonzalez, P., Kharin, V., Merryfield, W., Deser, C., Mason, S. J., Kirtman, B. P., Msadek, R., Sutton, R., Hawkins, E., Fricker, T., Hegerl, G., Ferro, C. A. T., Stephenson, D. B., Meehl, G. A., Stockdale, T., Burgman, R., Greene, A. M., Kushnir, Y., Newman, M., Carton, J., Fukumori, I., and Delworth, T.: A verification framework for interannual-to-decadal predictions experiments, Clim. Dynam., 40, 245–272, https://doi.org/10.1007/s00382-012-1481-2, 2013.
Hassler, B., Hoffman, F. M., Beadling, R., Blockley, E., Bo, H., Lee, J., Lembo, V., Lewis, J., Lu, J., Madaus, L., Malinina, E., Medeiros, B., Pokam, W., Scoccimarro, E., and Swaminathan, R.: Systematic Benchmarking of Climate Models: Methodologies, Applications, and New Directions, Rev. Geophys., 64, e2025RG000891, https://doi.org/10.1029/2025RG000891, 2026.
Hermanson, L., Smith, D., Seabrook, M., Bilbao, R., Doblas-Reyes, F., Tourigny, E., Lapin, V., Kharin, V. V., Merryfield, W. J., Sospedra-Alfonso, R., Athanasiadis, P., Nicoli, D., Gualdi, S., Dunstone, N., Eade, R., Scaife, A., Collier, M., O'Kane, T., Kitsios, V., Sandery, P., Pankatz, K., Früh, B., Pohlmann, H., Müller, W., Kataoka, T., Tatebe, H., Ishii, M., Imada, Y., Kruschke, T., Koenigk, T., Karami, M. P., Yang, S., Tian, T., Zhang, L., Delworth, T., Yang, X., Zeng, F., Wang, Y., Counillon, F., Keenlyside, N., Bethke, I., Lean, J., Luterbacher, J., Kolli, R. K., and Kumar, A.: WMO Global Annual to Decadal Climate Update: A Prediction for 2021–25, B. Am. Meteorol. Soc., 103, E1117–E1129, https://doi.org/10.1175/BAMS-D-20-0311.1, 2022.
Hersbach, H., Bell, B., Berrisford, P., Hirahara, S., Horányi, A., Muñoz-Sabater, J., Nicolas, J., Peubey, C., Radu, R., Schepers, D., Simmons, A., Soci, C., Abdalla, S., Abellan, X., Balsamo, G., Bechtold, P., Biavati, G., Bidlot, J., Bonavita, M., De Chiara, G., Dahlgren, P., Dee, D., Diamantakis, M., Dragani, R., Flemming, J., Forbes, R., Fuentes, M., Geer, A., Haimberger, L., Healy, S., Hogan, R. J., Hólm, E., Janisková, M., Keeley, S., Laloyaux, P., Lopez, P., Lupu, C., Radnoti, G., de Rosnay, P., Rozum, I., Vamborg, F., Villaume, S., and Thépaut, S.-N.: The ERA5 global reanalysis, Q. J. Roy. Meteor. Soc., 146, 1999–2049, https://doi.org/10.1002/qj.3803, 2020.
Huang, J., Ou, T., Chen, D., Luo, Y., and Zhao, Z.: The amplified Arctic warming in the recent decades may have been overestimated by CMIP5 models, Geophy. Res. Lett., 46, 13338–13345, https://doi.org/10.1029/2019GL084385, 2019.
Huffman, G. J., Adler, R. F., Behrangi, A., Bolvin, D. T., Nelkin, E. J., Gu, G., and Ehsani, M. R.: The New Version 3.2 Global Precipitation Climatology Project (GPCP) Monthly and Daily Precipitation Products, J. Climate, 36, 7635–7655, https://doi.org/10.1175/JCLI-D-23-0123.1, 2023.
IPCC: Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change, edited by: Stocker, T. F., Qin, D., Plattner, G.-K., Tignor, M., Allen, S. K., Boschung, J., Nauels, A., Xia, Y., Bex, V., and Midgley, P. M., Cambridge University Press, Cambridge, United Kingdom and New York, NY, USA, 1535 pp., https://doi.org/10.1017/CBO9781107415324, 2013.
Ivanova, D. P., Gleckler, P. J., Taylor, K. E., Durack, P. J., and Marvel, K. D.: Moving beyond the Total Sea Ice Extent in Gauging Model Biases, J. Climate, 29, 8965–8987, https://doi.org/10.1175/JCLI-D-16-0026.1, 2016.
Jun, S.-Y., Kim, J.-H., Choi, J., Kim, S.-J., Kim, B.-M., and An, S.-I.: The internal origin of the west-east asymmetry of Antarctic climate change, Sci. Adv., 6, eaaz1490, https://doi.org/10.1126/sciadv.aaz1490, 2020.
Keenlyside, N. S., Latif, M., Jungclaus, J., Kornblueh, L., and Roeckner, E.: Advancing decadal-scale climate prediction in the North Atlantic sector, Nature, 453, 84–88, https://doi.org/10.1038/nature06921, 2008.
Kharin, V. V., Boer, G. J., Merryfield, W. J., Scinocca, J. F., and Lee, W.-S.: Statistical adjustment of decadal predictions in a changing climate, Geophys. Res. Lett., 39, L19705, https://doi.org/10.1029/2012GL052647, 2012.
Klavans, J. M., Cane, M. A., Clement, A. C., and Murphy, L. N.: NAO predictability from external forcing in the late 20th century, NPJ Clim. Atmos. Sci., 4, 22, https://doi.org/10.1038/s41612-021-00177-8, 2021.
Lee, J., Gleckler, P. J., Ahn, M.-S., Ordonez, A., Ullrich, P. A., Sperber, K. R., Taylor, K. E., Planton, Y. Y., Guilyardi, E., Durack, P., Bonfils, C., Zelinka, M. D., Chao, L.-W., Dong, B., Doutriaux, C., Zhang, C., Vo, T., Boutte, J., Wehner, M. F., Pendergrass, A. G., Kim, D., Xue, Z., Wittenberg, A. T., and Krasting, J.: Systematic and objective evaluation of Earth system models: PCMDI Metrics Package (PMP) version 3, Geosci. Model Dev., 17, 3919–3948, https://doi.org/10.5194/gmd-17-3919-2024, 2024.
Lee, J., Chang, K., and Choi, J.: PCMDI Metrics Package DCPP Workflow and Metrics, Zenodo [code], https://doi.org/10.5281/zenodo.20789780, 2026.
Li, J.-L. F., Xu, K.-M., Richardson, M., Lee, W.-L., Jiang, J. H., Yu, J.-Y., Wang, Y.-H., Fetzer, E., Wang, L.-C., Stephens, G., and Liang, H.-C.: Annual and seasonal mean tropical and subtropical precipitation bias in CMIP5 and CMIP6 models, Environ. Res. Lett., 15, 124068, https://doi.org/10.1088/1748-9326/abc7dd, 2020.
Lin, Y., Dong, W., Zhang, M., Xie, Y., Xue, W., Huang, J., and Luo, Y.: Causes of model dry and warm bias over central U.S. and impact on climate projections. Nat. Commun., 8, 881, https://doi.org/10.1038/s41467-017-01040-2, 2017.
Ma, H.-Y., Siongco, A. C., Klein, S. A., Xie, S., Karspeck, A. R., Raeder, K., Anderson, J. L., Lee, J., Kirtman, B. P., Merryfield, W. J., Murakami, H., and Tribbia, J. J.: On the Correspondence between Seasonal Forecast Biases and Long-Term Climate Biases in Sea Surface Temperature, J. Climate, 34, 427–446, https://doi.org/10.1175/JCLI-D-20-0338.1, 2021.
Maher, N., Phillips, A. S., Deser, C., Wills, R. C. J., Lehner, F., Fasullo, J., Caron, J. M., Brunner, L., Beyerle, U., and Jeffree, J.: The updated Multi-Model Large Ensemble Archive and the Climate Variability Diagnostics Package: new tools for the study of climate variability and change, Geosci. Model Dev., 18, 6341–6365, https://doi.org/10.5194/gmd-18-6341-2025, 2025.
Maloney, E. D., Gettelman, A., Ming, Y., Neelin, J. D., Barrie, D., Mariotti, A., Chen, C.-C., Coleman, D. R. B., Kuo, Y.-H., Singh, B., Annamalai, H., Berg, A., Booth, J. F., Camargo, S. J., Dai, A., Gonzalez, A., Hafner, J., Jiang, X., Jing, X., Kim, D., Kumar, A., Moon, Y., Naud, C. M., Sobel, A. H., Suzuki, K., Wang, F., Wang, J., Wing, A. A., Xu, X., and Zhao, M.: Process-Oriented Evaluation of Climate and Weather Forecasting Models, B. Am. Meteorol. Soc., 100, 1665–1686, https://doi.org/10.1175/BAMS-D-18-0042.1, 2019.
Meehl, G. A., Hu, A., and Teng, H.: Initialized decadal prediction for transition to positive phase of the Interdecadal Pacific Oscillation, Nat. Commun., 7, 11718, https://doi.org/10.1038/ncomms11718, 2016.
Meehl, G. A., Richter, J. H., Teng, H., Capotondi, A., Cobb, K., Doblas-Reyes, F., Donat, M. G., England, M. H., Fyfe, J. C., Han, W., Kim, H., Kirtman, B. P., Kushnir, Y., Lovenduski, N. S., Mann, M. E., Merryfield, W. J., Nieves, V., Pegion, K., Rosenbloom, N., Sanchez, S. C., Scaife, A. A., Smith, D., Subramanian, A. C., Sun, L., Thompson, D., Ummenhofer, C. C., and Xie, S.-P.: Initialized Earth System prediction from subseasonal to decadal timescales, Nat. Rev. Earth Environ., 2, 340–357, https://doi.org/10.1038/s43017-021-00155-x, 2021.
Meehl, G. A., Teng, H., Smith, D., Yeager, S., Merryfield, W., Doblas-Reyes, F., and Glanville, A. A.: The effects of bias, drift, and trends in calculating anomalies for evaluating skill of seasonal-to-decadal initialized climate predictions, Clim. Dynam., 59, 3373–3389, https://doi.org/10.1007/s00382-022-06272-7, 2022.
Mueller, B. and Seneviratne, S. I.: Systematic land climate and evapotranspiration biases in CMIP5 simulations, Geophys. Res. Lett., 41, 128–134, https://doi.org/10.1002/2013GL058055, 2014.
Mulholland, D. P., Laloyaux, P., Haines, K., and Balmaseda, M. A.: Origin and Impact of Initialization Shocks in Coupled Atmosphere–Ocean Forecasts, Mon. Weather Rev., 143, 4631–4644, https://doi.org/10.1175/MWR-D-15-0076.1, 2015.
Nadiga, B. T., Verma, T., Weijer, W., and Urban, N. M.: Enhancing skill of initialized decadal predictions using a dynamic model of drift, Geophy. Res. Lett., 46, 9991–9999, https://doi.org/10.1029/2019GL084223, 2019.
O'Kane, T. J., Scaife, A. A., Kushnir, Y., Brookshaw, A., Buontempo, C., Carlin, D., Connell, R. K., Doblas-Reyes, F., Dunstone, N., Förster, K., Graça, A., Hobday, A. J., Kitsios, V., van der Laan, L., Lockwood, J., Merryfield, W. J., Paxian, A., Payne, M. R., Reader, M. C., Saville, G. R., Smith, D., Solaraju-Murali, B., Caltabiano, N., Carman, J., Hawkins, E., Keenlyside, N., Kumar, A., Matei, D., Pohlmann, H., Power, S., Raphael, M., Sparrow, M., and Wu, B.: Recent applications and potential of near-term (interannual to decadal) climate predictions, Front. Clim., 5, 1121626, https://doi.org/10.3389/fclim.2023.1121626, 2023.
Planton, Y. Y., Guilyardi, E., Wittenberg, A. T., Lee, J., Gleckler, P. J., Bayr, T., McGregor, S., McPhaden, M. J., Power, S., Roehrig, R., Vialard, J., and Voldoire, A.: Evaluating Climate Models with the CLIVAR 2020 ENSO Metrics Package, B. Am. Meteorol. Soc., 102, E193–E217, https://doi.org/10.1175/BAMS-D-19-0337.1, 2021.
Priestley, M. D. K., Ackerley, D., Catto, J. L., Hodges, K. I., McDonald, R. E., and Lee, R. W.: An Overview of the Extratropical Storm Tracks in CMIP6 Historical Simulations. J. Climate, 33, 6315–6343, https://doi.org/10.1175/JCLI-D-19-0928.1, 2020.
Purich, A., Cai, W., England, M. H., and Cowan, T.: Evidence for link between modelled trends in Antarctic sea ice and underestimated westerly wind changes, Nat. Commun., 7, 10409, https://doi.org/10.1038/ncomms10409, 2016.
Rayner, N. A., Parker, D. E., Horton, E. B., Folland, C. K., Alexander, L. V., Rowell, D. P., Kent, E. C., and Kaplan, A.: Global analyses of sea surface temperature, sea ice, and night marine air temperature since the late nineteenth century, J. Geophys. Res., 108, 4407, https://doi.org/10.1029/2002JD002670, 2003.
Samanta, D., Karnauskas, K. B., and Goodkin, N. F.: Tropical Pacific SST and ITCZ biases in climate models: Double trouble for future rainfall projections?, Geophy. Res. Lett., 46, 2242–2252, https://doi.org/10.1029/2018GL081363, 2019.
Sanchez-Gomez, E., Cassou, C., Ruprich-Robert, Y., Fernandez, E., and Terray, L.: Drift dynamics in a coupled model initialized for decadal forecasts, Clim. Dynam., 46, 1819–1840, https://doi.org/10.1007/s00382-015-2678-y, 2016.
Sato, K. and Simmonds, I.: Antarctic skin temperature warming related to enhanced downward longwave radiation associated with increased atmospheric advection of moisture and temperature, Environ. Res. Lett., 16, 064059, https://doi.org/10.1088/1748-9326/ac0211, 2021.
Schemm, S.: Toward eliminating the decades-old “too zonal and too equatorward” storm-track bias in climate models, J. Adv. Model. Earth Sy., 15, e2022MS003482, https://doi.org/10.1029/2022MS003482, 2023.
Screen, J. A., Audette, A., Blackport, R., Deser, C., England, M., Feldl, N., Gervais, M., Hay, S., Kushner, P. J., Liang, Y.-C., Msadek, R., Mudhar, R., Sigmond, M., Smith, D., Sun, L., and Yu, H.: Causes and consequences of Arctic amplification elucidated by coordinated multimodel experiments, Commun. Earth Environ., https://doi.org/10.1038/s43247-025-03052-z, 2025.
Smith, D. M., Eade, R., and Pohlmann, H.: A comparison of full-field and anomaly initialization for seasonal to decadal climate prediction, Clim. Dynam., 41, 3325–3338, https://doi.org/10.1007/s00382-013-1683-2, 2013.
Smith, D. M., Screen, J. A., Deser, C., Cohen, J., Fyfe, J. C., García-Serrano, J., Jung, T., Kattsov, V., Matei, D., Msadek, R., Peings, Y., Sigmond, M., Ukita, J., Yoon, J.-H., and Zhang, X.: The Polar Amplification Model Intercomparison Project (PAMIP) contribution to CMIP6: investigating the causes and consequences of polar amplification, Geosci. Model Dev., 12, 1139–1164, https://doi.org/10.5194/gmd-12-1139-2019, 2019.
Solaraju-Murali, B., Bojovic, D., Gonzalez-Reviriego, N., Nicodemou, A., Terrado, M., Caron, L.-P., and Doblas-Reyes, F.-J.: How decadal predictions entered the climate services arena: an example from the agriculture sector, Climate Services, 27, 100303, https://doi.org/10.1016/j.cliser.2022.100303, 2022.
Tian, B. and Dong, X.: The double-ITCZ bias in CMIP3, CMIP5, and CMIP6 models based on annual mean precipitation, Geophy. Res. Lett., 47, e2020GL087232, https://doi.org/10.1029/2020gl087232, 2020.
Weisheimer, A., Baker, L. H., Bröcker, J., Garfinkel, C. I., Hardiman, S. C., Hodson, D. L. R., Palmer, T. N., Robson, J. I., Scaife, A. A., Screen, J. A., Shepherd, T. G., Smith, D. M., and Sutton, R. T.: The Signal-to-Noise Paradox in Climate Forecasts: Revisiting Our Understanding and Identifying Future Priorities, B. Am. Meteorol. Soc., 105, E651–E659. https://doi.org/10.1175/BAMS-D-24-0019.1, 2024.
Xue, Y., Simon, S. M., Anderson, J. R., Pegion, P., Barton, N. P., Baggett, C. F., Stan, C., Johnson, N. C., Akella, S., Becker, E., Mehra, A., Olsen, M., Shevliakova, E., Whitaker, J. S., and Zhu, J.: Advancing NOAA's Subseasonal and Seasonal Applications and Enhancing Collaboration among Stakeholders, Modelers, and Researchers, B. Am. Meteorol. Soc., 106, E1295–E1302, https://doi.org/10.1175/BAMS-D-25-0060.1, 2025.
Ye, K. and Messori, G.: Inter-model spread in the wintertime Arctic amplification in the CMIP6 models and the important role of internal climate variability, Global Planet. Change, 204, 103543, https://doi.org/10.1016/j.gloplacha.2021.103543, 2021.
Yhang, Y.-B., Lim, C.-M., and Jeong, D.: APEC climate center multi-model ensemble dataset for seasonal climate prediction, Sci. Data, 12, 303, https://doi.org/10.1038/s41597-025-04643-3, 2025.
Zhang, L., Delworth, T. L., Yang, X., Gudgel, R. G., Jia, L., Vecchi, G. A., and Zeng, F.: Estimating Decadal Predictability for the Southern Ocean Using the GFDL CM2.1 Model, J. Climate, 30, 5187–5203, https://doi.org/10.1175/JCLI-D-16-0840.1, 2017.
Zhang, X., Liu, H., and Zhang, M.: Double ITCZ in coupled ocean-atmosphere models: From CMIP3 to CMIP5, Geophy. Res. Lett., 42, 8651–8659, https://doi.org/10.1002/2015GL065973, 2015.
Short summary
As climate risks grow, society needs reliable predictions for the coming years and decades. We developed a framework to collectively compare climate prediction systems and examine their performances on global temperature, rainfall, and sea ice. As a complementary to traditional analyses, our new framework offers tracking evolution of model performance in simulation time, helping scientists and stakeholders better understand strengths and limits of decadal climate prediction.
As climate risks grow, society needs reliable predictions for the coming years and decades. We...
