Articles | Volume 9, issue 3
Development and technical paper
17 Mar 2016
Development and technical paper |  | 17 Mar 2016

Application of all-relevant feature selection for the failure analysis of parameter-induced simulation crashes in climate models

Wiesław Paja, Mariusz Wrzesien, Rafał Niemiec, and Witold R. Rudnicki

Abstract. Climate models are extremely complex pieces of software. They reflect the best knowledge on the physical components of the climate; nevertheless, they contain several parameters, which are too weakly constrained by observations, and can potentially lead to a simulation crashing. Recently a study by Lucas et al. (2013) has shown that machine learning methods can be used for predicting which combinations of parameters can lead to the simulation crashing and hence which processes described by these parameters need refined analyses. In the current study we reanalyse the data set used in this research using different methodology. We confirm the main conclusion of the original study concerning the suitability of machine learning for the prediction of crashes. We show that only three of the eight parameters indicated in the original study as relevant for prediction of the crash are indeed strongly relevant, three others are relevant but redundant and two are not relevant at all. We also show that the variance due to the split of data between training and validation sets has a large influence both on the accuracy of predictions and on the relative importance of variables; hence only a cross-validated approach can deliver a robust prediction of performance and relevance of variables.

Short summary
Climate simulation crashes are caused by inadeqaute knowledge of parameters in models describing physical phenomena. We have performed reanalysis of the data on simulation crashes and have shown that they can be attributed to three parameters of the ocean model. This is a significant improvement over the original study, where crashes were attributed to eight parameters. Our results will help researchers to develop improved models.