Code accessibility and code quality across phases of the models of the Coupled Model Intercomparison Project

Michael García-Rodríguez, Javier Rodeiro-Iglesias, and Juan A. Añel

Download

Final revised paper (published on 20 Aug 2026)
Preprint (discussion started on 12 Mar 2026)

Interactive discussion

Status: closed

RC1:
'Comment on egusphere-2025-6108', Anonymous Referee #1, 24 Apr 2026

In this paper, the authors study the accessibility, understandability, and licensing of climate models used across all CMIP exercises. They found that 18 models out of 121 were accessible, with one being from CMIP3, ten from CMIP5, and seven from CMIP6. Then, with the accessible models, the authors performed code analysis. They found out that all but one were written in Fortran, the exception being NICAM.09. Finally, for the models written in Fortran, they scored the code using the FortranAnalyser, a static code analysis tool that grades Fortran code from zero to ten. Their findings show that the quality of the code has increased with the newer phases of CMIP.

This work is of great importance as it provides an overview of the accessibility of the climate models used in CMIP. Equally important is its discussion regarding the reproducibility of these experiments, which are extremely consequential for advancing our understanding of climate.

I have three critiques for the authors.

1) In section 5, "Code Analysis," I would like to have a clarification, in general terms, of what the FortranAnalyser analyzes to score the models. Moreover, if possible, give an intuition of what a high score code would be. This is because the authors claim that a score of 4.014 is "relatively high," but I have no reference to compare this value with.

2) In the annexed table, where all the models are present, there is overlapped text (page 18, page 20).
3) In the references, the last accessed text is written in Spanish.

Citation: https://doi.org/10.5194/egusphere-2025-6108-RC1
- AC1: 'Reply on RC1', Michael García Rodríguez, 04 May 2026
  
  We would like to thank the reviewer for the careful reading of the manuscript and for the constructive comments. We greatly appreciate the positive assessment of the relevance of this work for understanding the accessibility, reproducibility, and software practices of CMIP climate models. Below we respond to each point in detail.
  
  Comment 1
  
  We thank the reviewer for this comment and agree that further clarification is useful.
  
  A detailed and comprehensive description of the metrics, methodology, and scoring system implemented in FortranAnalyser is provided in a dedicated article published in Software Impacts (García‑Rodríguez et al., 2024). In that work, the reader can find the full technical explanation of what FortranAnalyser evaluates, including the complete set of static metrics, how they are computed, and how they are aggregated into a final normalized score ranging from 0 to 10. For this reason, the present manuscript includes only a high-level description and cites that companion paper for methodological details.
  With regard to the interpretation of the score, we agree that the expression “relatively high” requires additional context. This wording is not meant to imply a high absolute level of software quality in a general software engineering sense. Rather, it is used strictly in a comparative, empirical sense, relative to the distribution of scores obtained for the set of accessible CMIP models analyzed in this study.
  
  Specifically, although the theoretical scale of FortranAnalyser spans from 0 to 10, the scores obtained for CMIP models fall within a much narrower empirical range between 2.7 and 4.3, despite covering several CMIP phases. Within this observed range, a value of 4.014 lies in the upper part of the distribution and is therefore relatively high with respect to other operational CMIP models, including many from more recent phases. At the same time, the fact that no model approaches even half of the theoretical maximum score underlines one of the central conclusions of the study: namely, that there remains substantial room for improvement in the structural quality and maintainability of climate model code, even among the best-performing cases.
  
  We will revise the manuscript to better emphasize this relative interpretation and to explicitly clarify that the FortranAnalyser score should not be understood as an absolute measure of software excellence.
  
  Comment 2
  
  We thank the reviewer for pointing this out. The overlap in the annexed table is a formatting issue introduced during the compilation of the manuscript. We will carefully revise the layout and ensure that all tables are correctly rendered and fully legible in the revised version.
  
  Comment 3
  
  We appreciate the reviewer’s attention to this detail. We will revise the references section and ensure that all “last accessed” notes are consistently written in English, in accordance with the journal’s language standards.
  
  Once again, we thank the reviewer for the constructive feedback, which will help us improve the clarity and presentation of the manuscript.
  
  Citation: https://doi.org/10.5194/egusphere-2025-6108-AC1
RC2:
'Comment on egusphere-2025-6108', Anonymous Referee #2, 20 May 2026

This manuscript introduces a systematic evaluation of code accessibility and code quality across the CMIP phases. The authors tried to access the code for as many climate models as possible that participated in all past CMIP phases. After discussing the code accessibility, programming languages and code quality were examined. This study is very useful to explain the status quo, but also to identify needs for future CMIP phases.
I have three comments:
- At the beginning of Section 3 (L.89/99), I am confused about the different numbers. First, it is written about "59 out of the 262 models", and then "18 out of 121". What is the difference? Are the second numbers only about the subset of coupled models? But then in L.114, there are "59 out of the 121 couples models" mentioned. How is this connected? Maybe you can make that clearer.
- In Section 4 about the programming languages, it would be very interesting to add an outlook or expectation about changes when hybrid models are added to the pool of CMIP models.
- In Section 5, it would be helpful to have some examples of lacks in the code quality which are found by the analysis tool. And which parts or topics should be focused on primarily by the modelling groups to improve their model code?

Citation: https://doi.org/10.5194/egusphere-2025-6108-RC2
- AC2: 'Reply on RC2', Michael García Rodríguez, 04 Jun 2026
  
  We thank the reviewer for the time and effort dedicated to reading and assessing our manuscript. We greatly appreciate the constructive comments provided. Below, we provide detailed responses to each of the comments raised.
  
  Comment 1
  
  At the beginning of Section 3 (L.89/99), I am confused about the different numbers. First, it is written about "59 out of the 262 models", and then "18 out of 121". What is the difference? Are the second numbers only about the subset of coupled models? But then in L.114, there are "59 out of the 121 couples models" mentioned. How is this connected? Maybe you can make that clearer.
  
  Response
  
  In order to eliminate the confusion regarding the numbering of models when referring to coupled models and individual models, Section 3 has been revised to clarify and explain this numbering scheme. Specifically, lines 89 through 95 have been rewritten:
  
  "Following the attempts made, successful access was obtained for 59 of the 262 individual models identified across all CMIP phases (see Figure 1 and Table A1). Here, the total of 262 refers to all individual models considered in this study. Within this broader set, a subset of 121 corresponds specifically to coupled climate models, which represent the core systems used in CMIP experiments.
  Out of these coupled models, 18 were successfully recovered; one from CMIP3, ten from CMIP5, and seven from CMIP6. Thus, the 59 accessible individual models include these 18 coupled models as a subset. For CMIP1 and CMIP2, no model could be recovered, as their code had not been preserved over time. CMIP7 models were not included in this study because this is the current CMIP phase of the project, and active development and usage are still ongoing."
  
  Comment 2
  
  In Section 4 about the programming languages, it would be very interesting to add an outlook or expectation about changes when hybrid models are added to the pool of CMIP models.
  
  Response
  
  Section 4 has been modified by adding the following text about the outlook or expectation about changes when hybrid models are added to the pool of CMIP models (lines 149-162):
  
  "The emergence of hybrid modelling approaches, combining physics implemented to code with machine learning or artificial intelligence techniques, is expected to further increase this heterogeneity. Although such approaches have not yet been integrated within the CMIP phases analysed in this study, their incorporation is foreseeable in future developments. These hybrids models may introduce new programming ecosystems alongside established high performance languages like Fortran. From a software engineering perspective, this transition may significantly impact code structure, dependencies, and development workflows. In particular, machine learning components introduce additional layers of complexity related not only to the implemented code, but also to training data, calibration procedures, and evolving model configurations. This raises important challenges for reproducibility and traceability, as the behaviour of such components may depend on factors beyond the static source code itself. Consequently, evaluating future CMIP models may require extending current approaches to software quality assessment. Traditional static code analysis techniques, such as those employed in this study, may need to be complemented with methodologies capable of addressing data provenance, model training processes, and the interpretability of learned parameterisations. In this context, hybrid models could be more appropriately analysed as systems composed of heterogeneous components, combining multiple tools specialised to a programming language and tools specialised on the analysis of machine learning usage modelling paradigms in order to obtain an integrated assessment of the full system."
  
  Comment 3
  In Section 5, it would be helpful to have some examples of lacks in the code quality which are found by the analysis tool. And which parts or topics should be focused on primarily by the modelling groups to improve their model code?
  
  Response
  
  In the manuscript, we will briefly include representative examples of typical issues detected by the static analysis, such as overly large and poorly modularised routines, high structural complexity, inconsistent variable declarations, and limited documentation practices. We will also clarify the main areas where improvement efforts could be most effective, particularly in terms of modularisation, coding standards, documentation, and the integration of analysis tools into development workflows.
  
  At the same time, we note that a much more detailed discussion of these aspects is already available in our recent publication dedicated to the FortranAnalyser tool (García-Rodríguez et al., 2024), where both hypothetical examples and real-case improvements are presented and analysed in depth. To avoid redundancy while still addressing the reviewer’s request, we will include a concise summary in Section 5 together with a clearer reference to that work, where readers can find a more comprehensive treatment of the identified code quality issues and possible solutions.
  
  Citation: https://doi.org/10.5194/egusphere-2025-6108-AC2

Peer review completion

AR – Author's response | RR – Referee report | ED – Editor decision | EF – Editorial file upload

AR by Michael García Rodríguez on behalf of the Authors (04 Jun 2026) Author's response Author's tracked changes Manuscript

ED: Publish subject to minor revisions (review by editor) (22 Jun 2026) by Julia Hargreaves

AR by Michael García Rodríguez on behalf of the Authors (23 Jun 2026) Author's response Author's tracked changes Manuscript

ED: Publish subject to technical corrections (23 Jun 2026) by Julia Hargreaves

ED: Publish as is (13 Jul 2026) by David Ham (Executive editor)

AR by Michael García Rodríguez on behalf of the Authors (23 Jul 2026) Manuscript

Editorial statement

The Climate Modelling Intercomparison Project coordinates climate projections made by the world's flagship climate models which feed into reports such as the IPCC. It should remain a concern that even in the 6th version of the project, relatively few of the models have made their code accessible to other scientists, let alone the general public. Furthermore, the consideration of what may or may not constitute good code is of interest to all those involved in geoscientifc model development. This paper may come at an interesting point in time. Since their inception in the 1960s, prognostic climate models have always used fortran and similar languages to create algorithms which directly reproduce our basic geoscientific understanding of how the climate evolves, through differential equations. The authors propose that things may be evolving in a new direction, with the potential incorporation of machine learning and other forms of AI into future generations of the models. This will surely present new challenges to the concept and assessment of scientific reproducibility and code development.

Short summary

We studied how accessible and reliable the computer code behind major climate models has been over time. By reviewing different phases of the Coupled Model Intercomparison Project, we found improvements in transparency and coding practices, but also gaps that limit reproducibility. Our work suggests practical steps to make future climate research more open, traceable, and trustworthy for scientists and society.