Articles | Volume 15, issue 8
https://doi.org/10.5194/gmd-15-3183-2022
© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/gmd-15-3183-2022
© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Methods for assessment of models
| 
19 Apr 2022
Methods for assessment of models |
| 19 Apr 2022
| 19 Apr 2022
An ensemble-based statistical methodology to detect differences in weather and climate model executables
Christian Zeman
CORRESPONDING AUTHOR
Institute for Atmospheric and Climate Science, ETH Zurich, Switzerland
Christoph Schär
Institute for Atmospheric and Climate Science, ETH Zurich, Switzerland
Related authors
Andreas Franz Prein, Praveen Pothapakula, Christian Zeman, Morgane Lalonde, and Marius Rixen
EGUsphere, https://doi.org/10.5194/egusphere-2025-6414, https://doi.org/10.5194/egusphere-2025-6414, 2026
This preprint is open for discussion and under review for Geoscientific Model Development (GMD).
Short summary
Short summary
We produce one of the world's most detailed global weather and climate simulations, spanning 4 years and enabling the direct representation of storms rather than approximations. This allows the capture of dangerous events such as strong wind gusts, heavy rain, and powerful tropical and mid-latitude storms anywhere on Earth. Our results show major improvements over traditional climate models, but also reveal remaining challenges in representing large, organized storm systems in the tropics.
Hugo Banderier, Christian Zeman, David Leutwyler, Stefan Rüdisühli, and Christoph Schär
Geosci. Model Dev., 17, 5573–5586, https://doi.org/10.5194/gmd-17-5573-2024, https://doi.org/10.5194/gmd-17-5573-2024, 2024
Short summary
Short summary
We investigate the effects of reduced-precision arithmetic in a state-of-the-art regional climate model by studying the results of 10-year-long simulations. After this time, the results of the reduced precision and the standard implementation are hardly different. This should encourage the use of reduced precision in climate models to exploit the speedup and memory savings it brings. The methodology used in this work can help researchers verify reduced-precision implementations of their model.
Qinggang Gao, Christian Zeman, Jesus Vergara-Temprado, Daniela C. A. Lima, Peter Molnar, and Christoph Schär
Weather Clim. Dynam., 4, 189–211, https://doi.org/10.5194/wcd-4-189-2023, https://doi.org/10.5194/wcd-4-189-2023, 2023
Short summary
Short summary
We developed a vortex identification algorithm for realistic atmospheric simulations. The algorithm enabled us to obtain a climatology of vortex shedding from Madeira Island for a 10-year simulation period. This first objective climatological analysis of vortex streets shows consistency with observed atmospheric conditions. The analysis shows a pronounced annual cycle with an increasing vortex shedding rate from April to August and a sudden decrease in September.
Christian Zeman, Nils P. Wedi, Peter D. Dueben, Nikolina Ban, and Christoph Schär
Geosci. Model Dev., 14, 4617–4639, https://doi.org/10.5194/gmd-14-4617-2021, https://doi.org/10.5194/gmd-14-4617-2021, 2021
Short summary
Short summary
Kilometer-scale atmospheric models allow us to partially resolve thunderstorms and thus improve their representation. We present an intercomparison between two distinct atmospheric models for 2 summer days with heavy thunderstorms over Europe. We show the dependence of precipitation and vertical wind speed on spatial and temporal resolution and also discuss the possible influence of the system of equations, numerical methods, and diffusion in the models.
Anurag Dipankar, Mauro Bianco, Mona Bukenberger, Till Ehrengruber, Nicoletta Farabullini, Oliver Fuhrer, Abishek Gopal, Daniel Hupp, Andreas Jocksch, Samuel Kellerhals, Clarissa A. Kroll, Xavier Lapillonne, Matthieu Leclair, Magdalena Luz, Christoph Müller, Chia Rui Ong, Carlos Osuna, Praveen Pothapakula, Andreas Prein, Matthias Röthlin, William Sawyer, Christoph Schär, Sebastian Schemm, Giacomo Serafini, Hannes Vogt, Ben Weber, Robert C. Jnglin Wills, Nicolas Gruber, and Thomas C. Schulthess
Geosci. Model Dev., 19, 713–729, https://doi.org/10.5194/gmd-19-713-2026, https://doi.org/10.5194/gmd-19-713-2026, 2026
Short summary
Short summary
Climate models are becoming more detailed and accurate by simulating weather at scales of just a few kilometers. Simulating at km-scale is computationally demanding requiring powerful supercomputers and efficient code. This work presents a refactored dynamical core of a state-of-the-art climate model using a Python-based approach. The refactored code has passed through a sequence of verification and validation demonstrating its usability in performing km-scale global simulations.
Andreas Franz Prein, Praveen Pothapakula, Christian Zeman, Morgane Lalonde, and Marius Rixen
EGUsphere, https://doi.org/10.5194/egusphere-2025-6414, https://doi.org/10.5194/egusphere-2025-6414, 2026
This preprint is open for discussion and under review for Geoscientific Model Development (GMD).
Short summary
Short summary
We produce one of the world's most detailed global weather and climate simulations, spanning 4 years and enabling the direct representation of storms rather than approximations. This allows the capture of dangerous events such as strong wind gusts, heavy rain, and powerful tropical and mid-latitude storms anywhere on Earth. Our results show major improvements over traditional climate models, but also reveal remaining challenges in representing large, organized storm systems in the tropics.
Stefano Ubbiali, Christian Kühnlein, Christoph Schär, Linda Schlemmer, Thomas C. Schulthess, Michael Staneker, and Heini Wernli
Geosci. Model Dev., 18, 529–546, https://doi.org/10.5194/gmd-18-529-2025, https://doi.org/10.5194/gmd-18-529-2025, 2025
Short summary
Short summary
We explore a high-level programming model for porting numerical weather prediction (NWP) model codes to graphics processing units (GPUs). We present a Python rewrite with the domain-specific library GT4Py (GridTools for Python) of two renowned cloud microphysics schemes and the associated tangent-linear and adjoint algorithms. We find excellent portability, competitive GPU performance, robust execution on diverse computing architectures, and enhanced code maintainability and user productivity.
Hugo Banderier, Christian Zeman, David Leutwyler, Stefan Rüdisühli, and Christoph Schär
Geosci. Model Dev., 17, 5573–5586, https://doi.org/10.5194/gmd-17-5573-2024, https://doi.org/10.5194/gmd-17-5573-2024, 2024
Short summary
Short summary
We investigate the effects of reduced-precision arithmetic in a state-of-the-art regional climate model by studying the results of 10-year-long simulations. After this time, the results of the reduced precision and the standard implementation are hardly different. This should encourage the use of reduced precision in climate models to exploit the speedup and memory savings it brings. The methodology used in this work can help researchers verify reduced-precision implementations of their model.
Ruoyi Cui, Nikolina Ban, Marie-Estelle Demory, Raffael Aellig, Oliver Fuhrer, Jonas Jucker, Xavier Lapillonne, and Christoph Schär
Weather Clim. Dynam., 4, 905–926, https://doi.org/10.5194/wcd-4-905-2023, https://doi.org/10.5194/wcd-4-905-2023, 2023
Short summary
Short summary
Our study focuses on severe convective storms that occur over the Alpine-Adriatic region. By running simulations for eight real cases and evaluating them against available observations, we found our models did a good job of simulating total precipitation, hail, and lightning. Overall, this research identified important meteorological factors for hail and lightning, and the results indicate that both HAILCAST and LPI diagnostics are promising candidates for future climate research.
Eleonora Dallan, Francesco Marra, Giorgia Fosser, Marco Marani, Giuseppe Formetta, Christoph Schär, and Marco Borga
Hydrol. Earth Syst. Sci., 27, 1133–1149, https://doi.org/10.5194/hess-27-1133-2023, https://doi.org/10.5194/hess-27-1133-2023, 2023
Short summary
Short summary
Convection-permitting climate models could represent future changes in extreme short-duration precipitation, which is critical for risk management. We use a non-asymptotic statistical method to estimate extremes from 10 years of simulations in an orographically complex area. Despite overall good agreement with rain gauges, the observed decrease of hourly extremes with elevation is not fully represented by the model. Climate model adjustment methods should consider the role of orography.
Roman Brogli, Christoph Heim, Jonas Mensch, Silje Lund Sørland, and Christoph Schär
Geosci. Model Dev., 16, 907–926, https://doi.org/10.5194/gmd-16-907-2023, https://doi.org/10.5194/gmd-16-907-2023, 2023
Short summary
Short summary
The pseudo-global-warming (PGW) approach is a downscaling methodology that imposes the large-scale GCM-based climate change signal on the boundary conditions of a regional climate simulation. It offers several benefits in comparison to conventional downscaling. We present a detailed description of the methodology, provide companion software to facilitate the preparation of PGW simulations, and present validation and sensitivity studies.
Qinggang Gao, Christian Zeman, Jesus Vergara-Temprado, Daniela C. A. Lima, Peter Molnar, and Christoph Schär
Weather Clim. Dynam., 4, 189–211, https://doi.org/10.5194/wcd-4-189-2023, https://doi.org/10.5194/wcd-4-189-2023, 2023
Short summary
Short summary
We developed a vortex identification algorithm for realistic atmospheric simulations. The algorithm enabled us to obtain a climatology of vortex shedding from Madeira Island for a 10-year simulation period. This first objective climatological analysis of vortex streets shows consistency with observed atmospheric conditions. The analysis shows a pronounced annual cycle with an increasing vortex shedding rate from April to August and a sudden decrease in September.
Christian R. Steger, Benjamin Steger, and Christoph Schär
Geosci. Model Dev., 15, 6817–6840, https://doi.org/10.5194/gmd-15-6817-2022, https://doi.org/10.5194/gmd-15-6817-2022, 2022
Short summary
Short summary
Terrain horizon and sky view factor are crucial quantities for many geoscientific applications; e.g. they are used to account for effects of terrain on surface radiation in climate and land surface models. Because typical terrain horizon algorithms are inefficient for high-resolution (< 30 m) elevation data, we developed a new algorithm based on a ray-tracing library. A comparison with two conventional methods revealed both its high performance and its accuracy for complex terrain.
Roman Brogli, Silje Lund Sørland, Nico Kröner, and Christoph Schär
Weather Clim. Dynam., 2, 1093–1110, https://doi.org/10.5194/wcd-2-1093-2021, https://doi.org/10.5194/wcd-2-1093-2021, 2021
Short summary
Short summary
In a warmer future climate, climate simulations predict that some land areas will experience excessive warming during summer. We show that the excessive summer warming is related to the vertical distribution of warming within the atmosphere. In regions characterized by excessive warming, much of the warming occurs close to the surface. In other regions, most of the warming is redistributed to higher levels in the atmosphere, which weakens the surface warming.
Daniel Regenass, Linda Schlemmer, Elena Jahr, and Christoph Schär
Hydrol. Earth Syst. Sci. Discuss., https://doi.org/10.5194/hess-2021-426, https://doi.org/10.5194/hess-2021-426, 2021
Manuscript not accepted for further review
Short summary
Short summary
Weather and climate models need to represent the water cycle on land in order to provide accurate estimates of moisture and energy exchange between the land and the atmosphere. Infiltration of water into the soil is often modeled with an equation describing water transport in porous media. Here, we point out some challenges arising in the numerical solution of this equation and show the consequences for the representation of the water cycle in modern weather and climate models.
Silje Lund Sørland, Roman Brogli, Praveen Kumar Pothapakula, Emmanuele Russo, Jonas Van de Walle, Bodo Ahrens, Ivonne Anders, Edoardo Bucchignani, Edouard L. Davin, Marie-Estelle Demory, Alessandro Dosio, Hendrik Feldmann, Barbara Früh, Beate Geyer, Klaus Keuler, Donghyun Lee, Delei Li, Nicole P. M. van Lipzig, Seung-Ki Min, Hans-Jürgen Panitz, Burkhardt Rockel, Christoph Schär, Christian Steger, and Wim Thiery
Geosci. Model Dev., 14, 5125–5154, https://doi.org/10.5194/gmd-14-5125-2021, https://doi.org/10.5194/gmd-14-5125-2021, 2021
Short summary
Short summary
We review the contribution from the CLM-Community to regional climate projections following the CORDEX framework over Europe, South Asia, East Asia, Australasia, and Africa. How the model configuration, horizontal and vertical resolutions, and choice of driving data influence the model results for the five domains is assessed, with the purpose of aiding the planning and design of regional climate simulations in the future.
Christian Zeman, Nils P. Wedi, Peter D. Dueben, Nikolina Ban, and Christoph Schär
Geosci. Model Dev., 14, 4617–4639, https://doi.org/10.5194/gmd-14-4617-2021, https://doi.org/10.5194/gmd-14-4617-2021, 2021
Short summary
Short summary
Kilometer-scale atmospheric models allow us to partially resolve thunderstorms and thus improve their representation. We present an intercomparison between two distinct atmospheric models for 2 summer days with heavy thunderstorms over Europe. We show the dependence of precipitation and vertical wind speed on spatial and temporal resolution and also discuss the possible influence of the system of equations, numerical methods, and diffusion in the models.
Cited articles
Baker, A. H., Hammerling, D. M., Levy, M. N., Xu, H., Dennis, J. M., Eaton, B. E., Edwards, J., Hannay, C., Mickelson, S. A., Neale, R. B., Nychka, D., Shollenberger, J., Tribbia, J., Vertenstein, M., and Williamson, D.: A new ensemble-based consistency test for the Community Earth System Model (pyCECT v1.0), Geosci. Model Dev., 8, 2829–2840, https://doi.org/10.5194/gmd-8-2829-2015, 2015. a, b, c, d, e, f, g, h, i, j
Baker, A. H., Hu, Y., Hammerling, D. M., Tseng, Y.-H., Xu, H., Huang, X., Bryan, F. O., and Yang, G.: Evaluating statistical consistency in the ocean model component of the Community Earth System Model (pyCECT v2.0), Geosci. Model Dev., 9, 2391–2406, https://doi.org/10.5194/gmd-9-2391-2016, 2016. a, b, c, d
Baldauf, M., Seifert, A., Förstner, J., Majewski, D., Raschendorfer, M.,
and Reinhardt, T.: Operational Convective-Scale Numerical Weather Prediction
with the COSMO Model: Description and Sensitivities,
Mon. Weather Rev.,
139, 3887–3905, https://doi.org/10.1175/MWR-D-10-05013.1, 2011. a
Bartlett, M. S.: The Effect of Non-Normality on the t Distribution,
Math. Proc. Cambridge, 31,
223–231, https://doi.org/10.1017/S0305004100013311, 1935. a
Bauer, P., Thorpe, A., and Brunet, G.: The quiet revolution of numerical
weather prediction, Nature, 525, 47–55, https://doi.org/10.1038/nature14956, 2015. a
Bellprat, O., Kotlarski, S., Lüthi, D., De Elía, R., Frigon, A.,
Laprise, R., and Schär, C.: Objective calibration of regional climate
models: Application over Europe and North America, J. Climate, 29,
819–838, https://doi.org/10.1175/JCLI-D-15-0302.1, 2016. a
Benjamini, Y. and Hochberg, Y.: Controlling the False Discovery Rate: A
Practical and Powerful Approach to Multiple Testing,
J. Roy. Stat. Soc. B, 57, 289–300,
https://doi.org/10.1111/j.2517-6161.1995.tb02031.x, 1995. a, b, c, d
Carson, J. S.: Model verification and validation, in: Proceedings of the
Winter Simulation Conference, Winter Simulation Conference, San Diego, CA, USA, 8–11 December 2002, 1, 52–58,
https://doi.org/10.1109/WSC.2002.1172868, 2002. a
Clune, T. and Rood, R.: Software Testing and Verification in Climate Model
Development, IEEE Software, 28, 49–55, https://doi.org/10.1109/MS.2011.117, 2011. a
COSMO Consortium: COSMO Model License,
http://www.cosmo-model.org/content/consortium/licencing.htm,
last access: 12 April 2022. a
Dee, D. P., Uppala, S. M., Simmons, A. J., Berrisford, P., Poli, P., Kobayashi,
S., Andrae, U., Balmaseda, M. A., Balsamo, G., Bauer, P., Bechtold, P.,
Beljaars, A. C., van de Berg, L., Bidlot, J., Bormann, N., Delsol, C.,
Dragani, R., Fuentes, M., Geer, A. J., Haimberger, L., Healy, S. B.,
Hersbach, H., Hólm, E. V., Isaksen, L., Kållberg, P., Köhler,
M., Matricardi, M., Mcnally, A. P., Monge-Sanz, B. M., Morcrette, J. J.,
Park, B. K., Peubey, C., de Rosnay, P., Tavolato, C., Thépaut, J. N.,
and Vitart, F.: The ERA-Interim reanalysis: Configuration and performance of
the data assimilation system,
Q. J. Roy. Meteor. Soc., 137, 553–597, https://doi.org/10.1002/qj.828, 2011. a
Doms, G. and Baldauf, M.: A Description of the Nonhydrostatic Regional
COSMO-Model Part I: Dynamics and Numerics, Deutscher Wetterdienst (DWD), Offenbach, Germany,
https://doi.org/10.5676/DWD_pub/nwv/cosmo-doc_5.05_I, 2018. a, b
Doms, G., Förstner, J., Heise, E., Herzog, H.-J., Mironov, D.,
Raschendorfer, M., Reinhardt, T., Ritter, B., Schrodin, R., Schulz, J.-P.,
and Vogel, G.: COSMO Documentation Part II: Physical Parameterization,
Deutscher Wetterdienst (DWD), Offenbach, Germany, https://doi.org/10.5676/dwd_pub/nwv/cosmo-doc_5.05_ii, 2018. a
ECMWF: ERA-Interim reanalysis,
ECMWF [data set],
https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era-interim,
last access: 12 April 2022. a
Fuhrer, O., Osuna, C., Lapillonne, X., Gysi, T., Bianco, M., Arteaga, A., and
Schulthess, T. C.: Towards a performance portable, architecture agnostic
implementation strategy for weather and climate models,
Supercomputing Frontiers and Innovations, 1, 44–61, https://doi.org/10.14529/jsfi140103, 2014. a
Hong, S.-Y., Koo, M.-S., Jang, J., Kim, J.-E. E., Park, H., Joh, M.-S., Kang,
J.-H., and Oh, T.-J.: An Evaluation of the Software System Dependency of a
Global Atmospheric Model, Mon. Weather Rev., 141, 4165–4172,
https://doi.org/10.1175/MWR-D-12-00352.1, 2013. a
Knight, C. G., Knight, S. H. E., Massey, N., Aina, T., Christensen, C., Frame,
D. J., Kettleborough, J. A., Martin, A., Pascoe, S., Sanderson, B.,
Stainforth, D. A., and Allen, M. R.: Association of parameter, software, and
hardware variation with large-scale behavior across 57,000 climate models,
P. Natl. Acad. Sci. USA, 104, 12259–12264,
https://doi.org/10.1073/pnas.0608144104, 2007. a
Leutbecher, M. and Palmer, T. N.: Ensemble forecasting,
J. Comput. Phys., 227, 3515–3539,
https://doi.org/10.1016/j.jcp.2007.02.014, 2008. a
Livezey, R. E.: Statistical Analysis of General Circulation Model Climate
Simulation: Sensitivity and Prediction Experiments,
J. Atmos. Sci., 42, 1139–1150,
https://doi.org/10.1175/1520-0469(1985)042<1139:SAOGCM>2.0.CO;2, 1985. a
Livezey, R. E. and Chen, W. Y.: Statistical Field Significance and its
Determination by Monte Carlo Techniques, Mon. Weather Rev., 111,
46–59, https://doi.org/10.1175/1520-0493(1983)111<0046:SFSAID>2.0.CO;2, 1983. a, b
Lorenz, E. N.: Deterministic Nonperiodic Flow, J. Atmos. Sci., 20, 130–141, https://doi.org/10.1175/1520-0469(1963)020<0130:DNF>2.0.CO;2,
1963. a
Lott, F. and Miller, M. J.: A new subgrid-scale orographic drag
parametrization: Its formulation and testing,
Q. J. Roy. Meteor. Soc., 123, 101–127, https://doi.org/10.1256/smsqj.53703, 1997. a, b, c
Mahajan, S.: Ensuring Statistical Reproducibility of Ocean Model Simulations
in the Age of Hybrid Computing, in: Proceedings of the Platform for Advanced
Scientific Computing Conference, PASC '21, Association for Computing
Machinery, New York, NY, USA, 5–9 July 2021, https://doi.org/10.1145/3468267.3470572, 2021. a, b, c, d, e
Mahajan, S., Evans, K. J., Kennedy, J. H., Xu, M., and Norman, M. R.: A
Multivariate Approach to Ensure Statistical Reproducibility of Climate Model
Simulations, in: Proceedings of the Platform for Advanced Scientific
Computing Conference, PASC '19, Association for Computing Machinery, New
York, NY, USA, 12–14 June 2019, https://doi.org/10.1145/3324989.3325724, 2019. a, b, c
Mann, H. B. and Whitney, D. R.: On a Test of Whether one of Two Random
Variables is Stochastically Larger than the Other, Ann. Math. Stat., 18,
50–60, https://doi.org/10.1214/aoms/1177730491, 1947. a
Massonnet, F., Ménégoz, M., Acosta, M., Yepes-Arbós, X., Exarchou, E., and Doblas-Reyes, F. J.: Replicability of the EC-Earth3 Earth system model under a change in computing environment, Geosci. Model Dev., 13, 1165–1178, https://doi.org/10.5194/gmd-13-1165-2020, 2020. a, b, c
Oberkampf, W. L. and Roy, C. J.: Verification and Validation in Scientific
Computing, Cambridge University Press, https://doi.org/10.1017/CBO9780511760396, 2010. a
Oreskes, N.: Evaluation (not validation) of quantitative models,
Environ. Health Persp., 106, 1453–1460,
https://doi.org/10.1289/ehp.98106s61453, 1998. a
Oreskes, N., Shrader-Frechette, K., and Belitz, K.: Verification, Validation,
and Confirmation of Numerical Models in the Earth Sciences, Science, 263,
641–646, https://doi.org/10.1126/science.263.5147.641, 1994. a
Pithan, F., Angevine, W., and Mauritsen, T.: Improving a global model from the
boundary layer: Total turbulent energy and the neutral limit Prandtl number,
J. Adv. Model. Earth Sy., 7, 2029–2043,
https://doi.org/10.1002/2015MS000503, 2015. a
Posten, H. O.: Robustness of the Two-Sample T-Test, in: Robustness of
Statistical Methods and Nonparametric Statistics, edited by: Rasch, D. and
Tiku, M. L., Springer, Netherlands, Dordrecht,
92–99, https://doi.org/10.1007/978-94-009-6528-7_23, 1984. a
Raschendorfer, M.: The new turbulence parameterization of LM, COSMO
Newsletter, 1, 89–97,
http://www.cosmo-model.org/content/model/documentation/newsLetters/newsLetter01/newsLetter_01.pdf (last access: 9 April 2022),
2001. a
Reichler, T. and Kim, J.: How Well Do Coupled Models Simulate Today's
Climate?, B. Am. Meteorol. Soc., 89, 303–312,
https://doi.org/10.1175/BAMS-89-3-303, 2008. a
Reinhardt, T. and Seifert, A.: A three-category ice scheme for LMK, COSMO
Newsletter, 6, 115–120,
http://www.cosmo-model.org/content/model/documentation/newsLetters/newsLetter06/cnl6_reinhardt.pdf (last access: 9 April 2022),
2006. a
Ritter, B. and Geleyn, J.-F.: A Comprehensive Radiation Scheme for Numerical
Weather Prediction Models with Potential Applications in Climate
Simulations, Mon. Weather Rev., 120, 303–325, https://doi.org/10.1175/1520-0493(1992)120<0303:ACRSFN>2.0.CO;2, 1992. a
Rockel, B., Will, A., and Hense, A.: The regional climate model COSMO-CLM
(CCLM), Meteorol. Z., 17, 347–348,
https://doi.org/10.1127/0941-2948/2008/0309, 2008. a
Rosinski, J. M. and Williamson, D. L.: The Accumulation of Rounding Errors and
Port Validation for Global Atmospheric Models,
SIAM J. Sci. Comput., 18, 552–564, https://doi.org/10.1137/S1064827594275534, 1997. a, b
Sandu, I., Beljaars, A., Bechtold, P., Mauritsen, T., and Balsamo, G.: Why is
it so difficult to represent stably stratified conditions in numerical
weather prediction (NWP) models?, J. Adv. Model. Earth Sy., 5, 117–133, https://doi.org/10.1002/jame.20013, 2013. a
Sargent, R. G.: Verification and validation of simulation models,
J. Simul., 7, 12–24, https://doi.org/10.1057/jos.2012.20, 2013. a
Schär, C., Fuhrer, O., Arteaga, A., Ban, N., Charpilloz, C., Girolamo,
S. D., Hentgen, L., Hoefler, T., Lapillonne, X., Leutwyler, D., Osterried,
K., Panosetti, D., Rüdisühli, S., Schlemmer, L., Schulthess,
T. C., Sprenger, M., Ubbiali, S., and Wernli, H.: Kilometer-Scale Climate
Models, B. Am. Meteorol. Soc., 101, E567–E587,
https://doi.org/10.1175/BAMS-D-18-0167.1, 2020. a
Schättler, U., Doms, G., and Baldauf, M.: COSMO Documentation Part VII:
User's Guide, Deutscher Wetterdienst (DWD), Offenbach, Germany, https://doi.org/10.5676/dwd_pub/nwv/cosmo-doc_5.05_vii, 2018. a
Schlemmer, L., Schär, C., Lüthi, D., and Strebel, L.: A
Groundwater and Runoff Formulation for Weather and Climate Models, J. Adv. Model. Earth Sy., 10, 1809–1832,
https://doi.org/10.1029/2017MS001260, 2018. a
Storch, H. V.: A Remark on Chervin-Schneider's Algorithm to Test Significance
of Climate Experiments with GCM's, J. Atmos. Sci., 39,
187–189, https://doi.org/10.1175/1520-0469(1982)039<0187:AROCSA>2.0.CO;2, 1982. a
Student: The Probable Error of a Mean, Biometrika, 6, 1–25,
https://doi.org/10.2307/2331554, 1908. a
Sullivan, L. M. and D'Agostino, R. B.: Robustness of the t Test Applied to
Data Distorted from Normality by Floor Effects,
J. Dent. Res.,
71, 1938–1943, https://doi.org/10.1177/00220345920710121601, 1992. a
Thomas, S. J., Hacker, J. P., Desgagné, M., and Stull, R. B.: An Ensemble
Analysis of Forecast Errors Related to Floating Point Performance,
Weather Forecast., 17, 898–906,
https://doi.org/10.1175/1520-0434(2002)017<0898:AEAOFE>2.0.CO;2, 2002. a
Tiedtke, M.: A comprehensive mass flux scheme for cumulus parameterization in
large-scale models, Mon. Weather Rev., 117, 1779–1800,
https://doi.org/10.1175/1520-0493(1989)117<1779:ACMFSF>2.0.CO;2, 1989. a
Ventura, V., Paciorek, C. J., and Risbey, J. S.: Controlling the Proportion of
Falsely Rejected Hypotheses when Conducting Multiple Tests with
Climatological Data, J. Climate, 17, 4343–4356,
https://doi.org/10.1175/3199.1, 2004. a
Wan, H., Zhang, K., Rasch, P. J., Singh, B., Chen, X., and Edwards, J.: A new and inexpensive non-bit-for-bit solution reproducibility test based on time step convergence (TSC1.0), Geosci. Model Dev., 10, 537–552, https://doi.org/10.5194/gmd-10-537-2017, 2017. a
Whitner, R. B. and Balci, O.: Guidelines for Selecting and Using Simulation
Model Verification Techniques, in: Proceedings of the 21st Conference on
Winter Simulation, WSC '89, 4–6 December 1989, Association for Computing
Machinery, New York, NY, USA, 559–568, https://doi.org/10.1145/76738.76811, 1989. a, b, c
Wicker, L. J. and Skamarock, W. C.: Time-Splitting Methods for Elastic Models
Using Forward Time Schemes, Mon. Weather Rev., 130, 2088–2097,
https://doi.org/10.1175/1520-0493(2002)130<2088:TSMFEM>2.0.CO;2, 2002. a
Wilcox, R. R.: Some practical reasons for reconsidering the Kolmogorov-Smirnov
test, Brit. J. Math. Stat. Psy., 50, 9–20,
https://doi.org/10.1111/j.2044-8317.1997.tb01098.x, 1997. a
Wilks, D. S.: “The Stippling Shows Statistically Significant Grid Points”:
How Research Results are Routinely Overstated and Overinterpreted, and What
to Do about It, B. Am. Meteorol. Soc., 97,
2263–2273, https://doi.org/10.1175/BAMS-D-15-00267.1, 2016. a
Zadra, A., Roch, M., Laroche, S., and Charron, M.: The subgrid-scale
orographic blocking parametrization of the GEM Model, Atmos. Ocean,
41, 155–170, https://doi.org/10.3137/ao.410204, 2003.
a
Zeman, C. and Schär, C.: Data for “An Ensemble-Based Statistical Methodology to Detect Differences in Weather and Climate Model Executables” Part 1/2, Zenodo [data set], https://doi.org/10.5281/zenodo.6354200, 2021. a
Zeman, C. and Schär, C.: Data for “An Ensemble-Based Statistical Methodology to Detect Differences in Weather and Climate Model Executables” Part 2/2, Zenodo [data set], https://doi.org/10.5281/zenodo.6355647, 2022a. a
Zeman, C. and Schär, C.: Source Code for “An Ensemble-Based Statistical Methodology to Detect Differences in Weather and Climate Model Executables”, Zenodo [code], https://doi.org/10.5281/zenodo.6355694, 2022b. a
Zeman, C., Wedi, N. P., Dueben, P. D., Ban, N., and Schär, C.: Model intercomparison of COSMO 5.0 and IFS 45r1 at kilometer-scale grid spacing, Geosci. Model Dev., 14, 4617–4639, https://doi.org/10.5194/gmd-14-4617-2021, 2021. a
Zimmerman, D. W.: Comparative Power of Student T Test and Mann-Whitney U Test
for Unequal Sample Sizes and Variances,
J. Exp. Educ., 55, 171–174, https://doi.org/10.1080/00220973.1987.10806451, 1987. a
Short summary
Our atmosphere is a chaotic system, where even a tiny change can have a big impact. This makes it difficult to assess if small changes, such as the move to a new hardware architecture, will significantly affect a weather and climate model. We present a methodology that allows to objectively verify this. The methodology is applied to several test cases, showing a high sensitivity. Results also show that a major system update of the underlying supercomputer did not significantly affect our model.
Our atmosphere is a chaotic system, where even a tiny change can have a big impact. This makes...
