Articles | Volume 18, issue 8
https://doi.org/10.5194/gmd-18-2349-2025
© Author(s) 2025. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/gmd-18-2349-2025
© Author(s) 2025. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
The ensemble consistency test: from CESM to MPAS and beyond
Teo Price-Broncucia
CORRESPONDING AUTHOR
Department of Computer Science, University of Colorado, Boulder, Boulder, Colorado, USA
Allison Baker
NSF National Center for Atmospheric Research, Boulder, Colorado, USA
Dorit Hammerling
Department of Applied Mathematics and Statistics, Colorado School of Mines, Golden, Colorado, USA
Michael Duda
NSF National Center for Atmospheric Research, Boulder, Colorado, USA
Rebecca Morrison
Department of Computer Science, University of Colorado, Boulder, Boulder, Colorado, USA
Related authors
No articles found.
Finn Rebassoo, Peter Caldwell, Peter Lindstrom, Allison Baker, and Elynn Wu
EGUsphere, https://doi.org/10.5194/egusphere-2026-6708, https://doi.org/10.5194/egusphere-2026-6708, 2026
This preprint is open for discussion and under review for Geoscientific Model Development (GMD).
Short summary
Short summary
Machine-learning models can learn to approximate Earth system simulations and run them much faster. However, these models must be trained on large amounts of stored data, which grow enormously at higher resolution. This paper explores reducing the data needed for training using data compression. Training data compressed to one fifth the size of the original gave the same accuracy as the uncompressed training data. This could make training on high-resolution Earth system models more practical.
Tim Reichelt, Juniper Tyree, Milan Klöwer, Peter Dueben, Bryan N. Lawrence, Allison H. Baker, Sara Faghih-Naini, Torsten Hoefler, and Philip Stier
Geosci. Model Dev., 19, 5933–5960, https://doi.org/10.5194/gmd-19-5933-2026, https://doi.org/10.5194/gmd-19-5933-2026, 2026
Short summary
Short summary
The growing size of datasets used in climate science makes it difficult to store, analyze, and distribute dataset. Lossy compression algorithms can significantly reduce the disk space required to store datasets, but it can be difficult to understand and compare the behavior of different compression algorithms. ClimateBenchPress provides a benchmark to standardize comparisons between lossy compression algorithms and guide development of novel algorithms specifically targeted towards climate data.
Soyoung Ha, Jonathan J. Guerrette, Ivette Hernández Baños, William C. Skamarock, and Michael G. Duda
Geosci. Model Dev., 17, 4199–4211, https://doi.org/10.5194/gmd-17-4199-2024, https://doi.org/10.5194/gmd-17-4199-2024, 2024
Short summary
Short summary
To mitigate the imbalances in the initial conditions, this study introduces our recent implementation of the incremental analysis update (IAU) in the Model for Prediction Across Scales – Atmospheric (MPAS-A) component coupled with the Joint Effort for Data assimilation Integration (JEDI) through the cycling system. A month-long cycling run demonstrates the successful implementation of the IAU capability in the MPAS–JEDI cycling system.
Xingying Huang, Andrew Gettelman, William C. Skamarock, Peter Hjort Lauritzen, Miles Curry, Adam Herrington, John T. Truesdale, and Michael Duda
Geosci. Model Dev., 15, 8135–8151, https://doi.org/10.5194/gmd-15-8135-2022, https://doi.org/10.5194/gmd-15-8135-2022, 2022
Short summary
Short summary
We focus on the recent development of a state-of-the-art storm-resolving global climate model and investigate how this next-generation model performs for precipitation prediction over the western USA. Results show realistic representations of precipitation with significantly enhanced snowpack over complex terrains. The model evaluation advances the unified modeling of large-scale forcing constraints and realistic fine-scale features to advance multi-scale climate predictions and changes.
Cited articles
Ahn, D. H., Baker, A. H., Bentley, M., Briggs, I., Gopalakrishnan, G., Hammerling, D. M., Laguna, I., Lee, G. L., Milroy, D. J., and Vertenstein, M.: Keeping science on keel when software moves, Commun. ACM, 64, 66–74, https://doi.org/10.1145/3382037, 2021. a, b, c
Baker, A. H., Hammerling, D. M., Levy, M. N., Xu, H., Dennis, J. M., Eaton, B. E., Edwards, J., Hannay, C., Mickelson, S. A., Neale, R. B., Nychka, D., Shollenberger, J., Tribbia, J., Vertenstein, M., and Williamson, D.: A new ensemble-based consistency test for the Community Earth System Model (pyCECT v1.0), Geosci. Model Dev., 8, 2829–2840, https://doi.org/10.5194/gmd-8-2829-2015, 2015. a, b, c, d, e
Baker, A. H., Hu, Y., Hammerling, D. M., Tseng, Y.-H., Xu, H., Huang, X., Bryan, F. O., and Yang, G.: Evaluating statistical consistency in the ocean model component of the Community Earth System Model (pyCECT v2.0), Geosci. Model Dev., 9, 2391–2406, https://doi.org/10.5194/gmd-9-2391-2016, 2016. a
Baker, A., Price-Broncucia, T., Xu, H., Milroy, D., and Johnson, B.: pyCECT: Tools to support and run the CESM Ensemble Consistency Test, Zenodo [code and data set] https://doi.org/10.5281/zenodo.11662747, 2024.
Charney, J. G., Fjörtoft, R., and Von Neumann, J.: Numerical Integration of the Barotropic Vorticity Equation, Tellus, 2, 237–254, https://doi.org/10.1111/j.2153-3490.1950.tb00336.x, 1950. a
Cheng, X., Nitsche, G., and Wallace, J. M.: Robustness of Low-Frequency Circulation Patterns Derived from EOF and Rotated EOF Analyses, J. Climate, 8, 1709–1713, https://doi.org/10.1175/1520-0442(1995)008<1709:ROLFCP>2.0.CO;2, 1995. a
Clune, T. and Rood, R.: Software Testing and Verification in Climate Model Development, IEEE Software, 28, 49–55, https://doi.org/10.1109/MS.2011.117, 2011. a
Danabasoglu, G., Lamarque, J.-F., Bacmeister, J., Bailey, D. A., DuVivier, A. K., Edwards, J., Emmons, L. K., Fasullo, J., Garcia, R., Gettelman, A., Hannay, C., Holland, M. M., Large, W. G., Lauritzen, P. H., Lawrence, D. M., Lenaerts, J. T. M., Lindsay, K., Lipscomb, W. H., Mills, M. J., Neale, R., Oleson, K. W., Otto-Bliesner, B., Phillips, A. S., Sacks, W., Tilmes, S., van Kampenhout, L., Vertenstein, M., Bertini, A., Dennis, J., Deser, C., Fischer, C., Fox-Kemper, B., Kay, J. E., Kinnison, D., Kushner, P. J., Larson, V. E., Long, M. C., Mickelson, S., Moore, J. K., Nienhouse, E., Polvani, L., Rasch, P. J., and Strand, W. G.: The Community Earth System Model Version 2 (CESM2), J. Adv. Model. Earth Sy., 12, e2019MS001916, https://doi.org/10.1029/2019MS001916, 2020. a
Dijkstra, E. W.: Why numbering should start at zero, University of Texas, http://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF (last access: 3 April 2025), 1982. a
Easterbrook, S., Edwards, P., Balaji, V., and Budich, R.: Guest Editors' Introduction: Climate Change – Science and Software, IEEE Software, 28, 32–35, 2011. a
Easterbrook, S. M. and Johns, T. C.: Engineering the Software for Understanding Climate Change, Comput. Sci. Eng., 11, 65–74, https://doi.org/10.1109/MCSE.2009.193, 2009. a
ESCOMP: CAM: The Community Atmosphere Model, GitHub [code], https://github.com/ESCOMP/CAM, last access: 3 April 2025a.
ESCOMP: CESM: The Community Earth System Model, GitHub [code], https://github.com/ESCOMP/CESM, last access: 3 April 2025b.
Gokhale, M., Gopalakrishnan, G., Mayo, J., Nagarakatte, S., Rubio-González, C., and Siegel, S. F.: Report of the DOE/NSF Workshop on Correctness in Scientific Computing, June 2023, Orlando, FL, https://doi.org/10.48550/arXiv.2312.15640, 2023. a
Jackson, J. E. and Jackson, J. E.: A user's guide to principal components, Wiley series in probability and mathematical statistics, 1. print edn., Wiley, New York, https://doi.org/10.1002/0471725331, 1991. a, b
Krock, M. L., Kleiber, W., Hammerling, D., and Becker, S.: Modeling Massive Highly Multivariate Nonstationary Spatial Data with the Basis Graphical Lasso, J. Comput. Graph. Stat., 32, 1472–1487, https://doi.org/10.1080/10618600.2023.2174126, 2023. a
Lawley, D. N.: Tests of Significance for the Latent Roots of Covariance and Correlation Matrices, Oxford University Press, Biometrika Trust, 128–136, https://doi.org/10.2307/2333586, 1956. a
Mahajan, S.: Ensuring statistical reproducibility of ocean model simulations in the age of hybrid computing, in: Proceedings of the Platform for Advanced Scientific Computing Conference, PASC '21, Geneva, Switzerland, 5–9 July 2021, Association for Computing Machinery, New York, NY, USA, https://doi.org/10.1145/3468267.3470572, pp. 1–9, 2021. a, b
Mahajan, S., Gaddis, A. L., Evans, K. J., and Norman, M. R.: Exploring an Ensemble-Based Approach to Atmospheric Climate Modeling and Testing at Scale, Procedia Comput. Sci., 108, 735–744, https://doi.org/10.1016/j.procs.2017.05.259, 2017. a
Mahajan, S., Evans, K. J., Kennedy, J. H., Xu, M., and Norman, M. R.: A Multivariate Approach to Ensure Statistical Reproducibility of Climate Model Simulations, in: Proceedings of the Platform for Advanced Scientific Computing Conference, Zurich, Switzerland, 12–14 June 2019, ACM, Zurich Switzerland, https://doi.org/10.1145/3324989.3325724, pp. 1–10, 2019a. a
Mahajan, S., Evans, K. J., Kennedy, J. H., Xu, M., Norman, M. R., and Branstetter, M. L.: Ongoing solution reproducibility of earth system models as they progress toward exascale computing, Int. J. High Perform. C., 33, 784–790, https://doi.org/10.1177/1094342019837341, 2019b. a
Massonnet, F., Ménégoz, M., Acosta, M., Yepes-Arbós, X., Exarchou, E., and Doblas-Reyes, F. J.: Replicability of the EC-Earth3 Earth system model under a change in computing environment, Geosci. Model Dev., 13, 1165–1178, https://doi.org/10.5194/gmd-13-1165-2020, 2020. a
Milroy, D. J., Baker, A. H., Hammerling, D. M., Dennis, J. M., Mickelson, S. A., and Jessup, E. R.: Towards Characterizing the Variability of Statistically Consistent Community Earth System Model Simulations, Procedia Comput. Sci., 80, 1589–1600, https://doi.org/10.1016/j.procs.2016.05.489, 2016. a, b
Milroy, D. J., Baker, A. H., Hammerling, D. M., Kim, Y., Jessup, E. R., and Hauser, T.: Making Root Cause Analysis Feasible for Large Code Bases: A Solution Approach for a Climate Model, in: Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing, HPDC '19, Phoenix, AZ, USA, 22–29 June 2019, Association for Computing Machinery, New York, NY, USA, https://doi.org/10.1145/3307681.3325399, p. 73–84, 2019. a
Molinari, S., Milroy, D., and Hammerling, D.: A Statistical Investigation of the CESM Ensemble Consistency Testing Framework – Part II, Tech. rep., NSF National Center for Atmospheric Research, https://doi.org/10.5065/Y541-X174, 2024. a
Molinari, S. J., Milroy, D. J., and Hammerling, D. M.: A Statistical Investigation of the CESM Ensemble Consistency Testing Framework, NCAR Technical Note (No. NCAR/TN-554+STR), National Center for Atmospheric Research, https://doi.org/10.26024/bfdr-nz31, 2018. a, b, c
MPAS-Dev: MPAS-Model: Model for Prediction Across Scales models and shared framework releases, GitHub [code], https://github.com/MPAS-Dev/MPAS-Model, last access: 3 April 2025.
NCAR: Cheyenne Supercomputer, National Center for Atmospheric Research, https://doi.org/10.5065/D6RX99HX, 2017.
NCAR: Derecho Supercomputer, National Center for Atmospheric Research, https://doi.org/10.5065/qx9a-pg09, 2024.
North, G. R., Bell, T. L., Cahalan, R. F., and Moeng, F. J.: Sampling Errors in the Estimation of Empirical Orthogonal Functions, Mon. Weather Rev., 110, 699–706, https://doi.org/10.1175/1520-0493(1982)110<0699:SEITEO>2.0.CO;2, 1982. a
Oberkampf, W. L. and Trucano, T. G.: Verification and validation benchmarks, Nucl. Eng. Des., 238, 716–743, https://doi.org/10.1016/j.nucengdes.2007.02.032, 2008. a
Pollio, V.: Vitruvius: ten books on architecture/translated by Ingrid D. Rowland; commentary and illustrations by Thomas Noble Howe, with additional commentary by: Ingrid D. Rowland and Michael J. Dewar, Cambridge University Press, ISBN 0521553644, 1999. a
Richman, M. B. and Lamb, P. J.: Climatic Pattern Analysis of Three- and Seven-Day Summer Rainfall in the Central United States: Some Methodological Considerations and a Regionalization, J. Appl. Meteorol. Clim., 24, 1325–1343, https://doi.org/10.1175/1520-0450(1985)024<1325:CPAOTA>2.0.CO;2, 1985. a
Rosinski, J. M. and Williamson, D. L.: The accumulation of rounding errors and port validation for global atmospheric models, SIAM J. Sci. Comput., 18, 552–564, https://doi.org/10.1137/S1064827594275534, 1997. a, b
Shapiro, S. S. and Wilk, M. B.: An analysis of variance test for normality (complete samples), Biometrika, 52, 591–611, https://doi.org/10.1093/biomet/52.3-4.591, 1965. a
Skamarock, W. C., Klemp, J. B., Duda, M. G., Fowler, L. D., Park, S.-H., and Ringler, T. D.: A Multiscale Nonhydrostatic Atmospheric Model Using Centroidal Voronoi Tesselations and C-Grid Staggering, Mon. Weather Rev., 140, 3090–3105, https://doi.org/10.1175/MWR-D-11-00215.1, 2012. a, b
Wan, H., Zhang, K., Rasch, P. J., Singh, B., Chen, X., and Edwards, J.: A new and inexpensive non-bit-for-bit solution reproducibility test based on time step convergence (TSC1.0), Geosci. Model Dev., 10, 537–552, https://doi.org/10.5194/gmd-10-537-2017, 2017. a, b
Short summary
The ensemble consistency test (ECT) and its ultrafast variant (UF-ECT) have become powerful tools in the development community for the identification of unwanted changes in the Community Earth System Model (CESM). We develop a generalized setup framework to enable easy adoption of the ECT approach for other model developers and communities. This framework specifies test parameters to accurately characterize model variability and balance test sensitivity and computational cost.
The ensemble consistency test (ECT) and its ultrafast variant (UF-ECT) have become powerful...
Altmetrics
Final-revised paper
Preprint