Reproducibility is a cornerstone of science and the scientific method, and the challenge of achieving reproducible science when computational methods are involved is well known . In recent years, it has been a topic that is recurrently discussed by the scientific community (e.g., ).

Ensuring computational scientific reproducibility (CSR) requires addressing a variety of issues when designing and using models, particularly in climate models . Among these issues are legal aspects of software distribution and intellectual property, which are often unknown to researchers. Several studies have revealed very low levels of CSR . Efforts to address the problem have included informal initiatives, such as documenting the accessibility of climate models , as well as more formal assessments that evaluate model quality and transparency (e.g., ).

Improving the CSR is possible through a range of actions. For example, journals may adopt code and data policies , while resarchers can follow recommended practices such as providing complete software documentation, structuring code into functions or modules, and publishing code and data in trusted repositories that ensure long-term preservation with permanent identifiers such as a Digital Object Identifier (DOI) .

In some research fields, it is common practice to publish detailed methodological information. However, this habit is far less widespread in the development, maintenance, and scientific reproducibility of climate models . It remains concerning that the underlying model code is often not available, especially given its fundamental role in scientific research. Beyond basic scientific transparency, there are additional reasons to support the availability of climate models code, including the need to preserve knowledge about the development cycles of these models. Moreover, previous studies have shown that climate models frequently lack adequate documentation .

Climate models have evolved over decades, from early attempts that simulated basic processes (radiative balance, convection, terrestrial rotation, etc.) to comprehensive systems that represent interactions among major subsystems such as the atmosphere, ocean, and land. In many cases, these early models gradually developed into the sophisticated models used today. Originally, research and development of models was undertaken independently by research groups around the world. However, in 1995, a collaborative approach emerged that aimed to share progress and results among groups, thus contributing to a better understanding of the problem of global climate modelling and climate change. This initiative became the “Coupled Model Intercomparison Project” (CMIP), which is now in its seventh phase (CMIP7). Over the years, CMIP has undergone several phases, and its evolution has raised several methodological questions, some of which had previously been discussed in other research fields, and adapted here to climate modelling. These include to define standardized experimental protocols, how to ensure reproducibility of simulations, or how to manage uncertainty and ensemble approaches.

In parallel with the development of climate models, software design practices have also advanced. Since the early days of software, code quality has been a major concern, and numerous approaches to asses it were proposed more than forty-five years ago (e.g. ). While many of the early ideas about software quality are now outdated due to changes in computer architectures, software quality metrics have continued to evolve to address current needs.

More recently, the quality of software used in science and engineering has become a critical issue, given its direct impact on reproducibility and replicability . Ensuring that scientific results can be reproduced requires not only high-quality code but also its availability and proper documentation. The lack of formal programming training among scientists, often leading to software-related challenges , has long been recognized as a barrier to achieving this goal. Nevertheless, scientific software development has improved as researchers adopt best practices from broader software engineering fields .

Previous studies have shown that scientific software still has substantial room for improvement . Consequently, several authors have proposed best practices to enhance scientific software quality, development workflows, and code accessibility . However, the issue remains complex. Early software development faced strict hardware limitations, which required careful resource optimization. Over time, exponential increases in computing power have often masked the consequences of suboptimal code . With the widespread use of outsourced computing resources and cloud technologies in scientific research and climate modelling , where resources may appear virtually unlimited, the perceived impact of inefficient code is diminished. For instance, climate modelling experiments often run on supercomputers for months and consume vast computational resources, which may unintentionally reinforce this perception.

Given these challenges, this work provides an analysis of the current state of the art of the accessibility and static code quality of the CMIP models across all their historical phases, from CMIP1 to CMIP6. This study complements existing research in the field and extends our previous efforts focused on models from the fifth phase of the CMIP (CMIP5) , a broader analysis repeatedly requested by some of our research peers in the climate modelling community. Section 2 describes the Methods used in this work, Sect. 3 evaluates the accessibility to the model code, Sect. 4 examines the programming language used in CMIP models, Sect. 5 presents results from static code analysis for the different models, and finally we offer a discussion and conclusions.

To verify the availability of the models, we employed a systematic approach similar to that described in . Our first step consisted of checking access to the source code of the model, its configuration, and the associated documentation, including details on parameters, input data, and other relevant components. We primarily relied on the official download links provided on the CMIP website. When direct access was not available, we search for contact information through institutional websites or broader Internet searches. When necessary, we attempted to communicate the responsible institution or researchers via email and telephone. If access request for a given model's code was denied, we followed up with a survey to better understand the reasons for the refusal. Unfortunately, we did not receive any responses. For models developed during the earliest CMIP phases, obtaining access was often impossible, as many of these versions had not been adequately preserved.

As in , this process revealed several barriers to transparency and reproducibility. For example, in some cases, the only available contact information was found in publications behind paywalls or embedded within metadata of NetCDF output files. Accessing such metadata requires technical skills and computational resources that are often beyond the reach of the general public and even some scientists.

For each climate model obtained, the licences associated with the model and potential restrictions to its use were verified. We also verified whether the model distribution included practical information on how to compile, configure, and execute the code to run an experiment. From a reproducibility perspective, this type of documentation is a very important aspect.

Furthermore, we analyse the evolution of the code quality across CMIP phases. To do so, we applied static code analysis, a method for evaluating programs without executing or compiling them. Specifically, we used the Fortran static code analysis tool FortranAnalyser v2.0 (http://fortrananalyser.ephyslab.uvigo.es/, last access: 31 July 2026) . We selected this tool because Fortran is the dominant programming language in the CMIP models (as demonstrated later), and because alternative tools have limited applicability for assessing scientific code written in Fortran .

FortranAnalyser evaluates code quality using metrics that can be grouped into two scoring categories. Metrics rated on a 0 to 2.0 scale include the use of implicit none, the presence and quality of comments across different code elements (files, variables, functions, subroutines, and control structures), the number of nested loops (as an indicator of structural complexity), and the comment-to-code ratio. In contrast, metrics rated on a 0 to 1.0 scale focus on specific control flow practices, namely the use of the exit and cycle statements. Together, these grouped metrics provide a balanced assessment of code readability, maintainability, and structural quality. FortranAnalyser gives additional information too; number of lines, the cyclomatic complexity, the number of functions declared, the number of variables declared, and the number of calls to subroutines among others. According to the accomplishment of these metrics, FortranAnalyser assigns a score from 0 to 10 (where 0 is the minimum score and 10 is the maximum) to each file analysed and generate a PDF quality report with the results.

Following the attempts made, successful access was obtained for 59 of the 262 individual models identified across all CMIP phases (see Fig. and Table ). Here, the total of 262 refers to all individual models considered in this study. Within this broader set, a subset of 121 corresponds specifically to coupled climate models, which represent the core systems used in CMIP experiments. Out of these coupled models, 18 were successfully recovered; one from CMIP3, ten from CMIP5, and seven from CMIP6. Thus, the 59 accessible individual models include these 18 coupled models as a subset. For CMIP1 and CMIP2, no model could be recovered, as their code had not been preserved over time. CMIP7 models were not included in this study because this is the current CMIP phase of the project, and active development and usage are still ongoing.

A pronounced regional bias emerged in the geographic distribution of accessible models. The United States of America, France, Germany, and Norway stand out as the most open contributors; in all cases, institutions granted access to the full set of models they developed, independently of the CMIP phase. This, however, remained conditional on the availability of older model versions that institutions had effectively preserved. We hypothesize that national or regional legalisation related to software copyright, intellectual property, and code disclosure may partly explain these differences.

Regarding previous analyses of CMIP5 accessibility, despite the fact that software can be patented in the United States, meanwhile it is not possible in the the European Union , we were able to obtain the source code for 10 out of the 28 coupled models contributed by U.S.-based research centres. In contrast, only 8 out of 34 EU-based models were obtained.

The potential explanations for this have not changed since the CMIP5 study. The involvement of U.S. federal employees in the development of some models may facilitate code release, as federal employees are legally required to make their work publicly available without restrictions . Similarly, the development of NorESM, which incorporates substantial components from the U.S.-developed model CESM1 , may have benefited from inherited code licensing, helping explain Norway's comparatively high accessibility rates. Conversely, decisions not to release model code may stem from legacy licensing constraints or the absence of copyright transfer agreements, which can prevent redistribution of older codebases.

To better visualize the regional distribution of accessible models, and illustrate differences in national policies governing code sharing, Fig. displays the percentage of models obtained per country. For models developed by EC-Earth and ECMWF, the corresponding data are displayed at a central European location (Slovakia), as these consortia involve between 12 and 23 European countries depending on the CMIP phase.

The current level accessibility to CMIP source codes remains critically limited. Only 59 out of the 121 coupled models were directly accessible; access to the remainder required active interaction, such as email correspondence with developers. As in previous work, even after identifying ourselves as researchers (addressing the potential concern that scientists might hesitate to share code with non-scientists), more than half of all model versions remain inaccessible.

There are several plausible reasons for limited code sharing among scientists and development teams. Some groups may face institutional or national legal restrictions preventing them from openly distributing source code. Others may be concerned about inheriting sharing rights for portions of the model code that were developed by different institutions or research groups, raising concerns about potential infringement on proprietary rights. In other cases, older models may no longer be available simply due to insufficient long-term maintenance of archives or repositories. In particular, when source code was obtained, the developers did not provide explanations for their licensing choices; indeed, in some cases, the code lacked a clear license defining usage terms.

Although there is no evidence suggesting that scientific results derived from CMIP models are invalid, the lack of code availability can be misused to cast doubt on the reproducibility of the underlying scientific findings. For this reason, we strongly encourage modelling groups to adopt more robust practices that align with modern expectations of corporate social responsibility in science, particularly regarding reproducibility and replicability (CSR). Previous work has shown that sharing model code structures, even if currently suboptimal, can promote improvements in software quality and scientific transparency .

Over the course of different CMIP phases, progressive improvements in spatial resolution, numerical accuracy, and the inclusion of crucial climate processes have been incorporated into the contributing models . These advances have involved the continuous refinement of computational algorithms, the addition of new functionalities, and the systematic replacement of outdated components, all aimed at enhancing the scientific robustness and overall quality of model outputs.

In this context, understanding how models and their successive versions operate, involve across CMIP phases, and adhere to established software development standards requires careful examination of the programming languages employed in their implementation. Programming languages play a role in defining the structure, performance, and maintainability of climate models. Table summarises the distribution of programming languages used across the models for which source code was available.

The choice of programming language in climate model development is strongly influenced by the scientific community behind each model, its computational requirements, and the degree of code inheritance involved. Climate models often emerge from long-term collaborative efforts involving multiple institutions and scientists disciplines, which can lead to the coexistence of multiple languages in a single project. While performance optimisation and scalability are essential considerations, given the high computational cost of climate simulations, such priorities are sometimes overridden by the need for rapid scientific deliverable.

The emergence of hybrid modelling approaches, combining physics implemented to code with machine learning or artificial intelligence techniques, is expected to further increase this heterogeneity. Although such approaches have not yet been integrated within the CMIP phases analysed in this study, their incorporation is foreseeable in future developments. These hybrids models may introduce new programming ecosystems alongside established high performance languages like Fortran. From a software engineering perspective, this transition may significantly impact code structure, dependencies, and development workflows. In particular, machine learning components introduce additional layers of complexity related not only to the implemented code, but also to training data, calibration procedures, and evolving model configurations. This raises important challenges for reproducibility and traceability, as the behaviour of such components may depend on factors beyond the static source code itself. Consequently, evaluating future CMIP models may require extending current approaches to software quality assessment. Traditional static code analysis techniques, such as those employed in this study, may need to be complemented with methodologies capable of addressing data provenance, model training processes, and the interpretability of learned parameterisations. In this context, hybrid models could be more appropriately analysed as systems composed of heterogeneous components, combining multiple tools specialised to a programming language and tools specialised on the analysis of machine learning usage modelling paradigms in order to obtain an integrated assessment of the full system.

To analyse the use of languages in the retrieved CMIP models, we employed the GitLab's repository analytics functionality , based on the GitHub “linguist” library . This functionality excludes non-relevant files and calculates the percentages of code used.

As shown in Table , Fortran remains the dominant programming language in CMIP models. Its prevalence is well documented and reflects its efficiency in handling computationally intensive scientific calculations, as well as its long-standing stability in large legacy projects. The substantial multi-institutional investment in these models, often comprising millions of lines of code across multiple components has made transitions to alternative languages both difficult and improbable. Consequently, Fortan has remained the primary choice to ensure consistency, compatibility, and continuity across successive model generations, with developers extending codebases rather than rewriting them.

Python has increasing prominence in the climate modelling community, particularly for tasks such as as pre and post processing of model output, workflow scripting, rapid prototyping, and visualisation. Scripting languages such as as Perl and Bash are also used for workflow automation, job orchestration on HPC systems, and data management tasks. Additionally, R is frequently employed for statistical analysis and evaluation of observational and model generated datasets.

C and C++ , are commonly used to optimise numerically demanding components or integrate specific modules requiring high performance. They often used alongside Fortran, either to improve the computational performance of specific subroutines or for compatibility with the use of other codes on which they depend. Notably, one of the models, CFSv2-2011, stands out as the only climate model in which C constitues the primary implementation language, accounting for 88.1 % of its codebase.

An additional noteworthy case is that NICAM.09 has been developed predominantly in CLisp . Nevertheless, Fortran remains the second most used language within NICAM.09, supporting its core dynamics and physics routines.

Based on these results and the overwhelming dominance of Fortran across the CMIP model suite, the present study focuses specifically on analysing the Fortran components of the obtained codebases.

Here, we focus exclusively on the analysis of Fortran code, reflecting its longstanding and central role in climate model development. This choice is supported by the results in Table , which shows how Fortran constitutes a substantial portion of the programming languages used across the CMIP models.

To evaluate code quality, we employed FortranAnalyser, a static analysis tool designed to assess software written in any Fortran version. Its compatibility with legacy code is particularly important given the extensive historical layers present in climate models. Moreover, FortranAnalyser was developed ad-hoc for this purposes, addressing the lack of alternative tools capable of performing consistent, large scale static analysis on Fortran scientific software. An example of improved Fortran code by applying FortranAnalyser recommendations is the case of the IPSL climate model, which improved its quality after an assessment performed with the tool. The case study is described in . Table summarizes the static code quality scores produced by FortranAnalyser for each CMIP model analysed. These scores are reported on a normalized scale from 0 to 10; however, it is important to note that the values observed for operational CMIP models occupy a much narrower empirical range. Across all analysed models and CMIP phases, scores span approximately from 2.7 to 4.3, indicating that even the best-performing climate models remain far from the theoretical maximum and highlighting substantial room for improvement in code structure and maintainability across CMIP phases.

In CMIP Phase 3, only one model, CCSM3, was available, yielding a score of 4.014. Within the empirical distribution observed across all CMIP models, this value lies in the upper part of the range and is therefore relatively high in a comparative sense. Nonetheless, the limited number of available models in this phase restricts the contextual interpretation of this result.

Moving to Phase 5, the number of evaluated models increases significantly, yielding a wider spread of the code quality. Models such as GISS-E2 (2.774) and CFSv2-2011 (2.920) score relatively low, indicating potential structural or maintainability weaknesses in their codebases. In contrast, NICAM.09 (4.341) and GEOS-5 (4.131) show stronger performance, highlighting heterogeneity in coding practices across modelling groups during this phase.

A particularly notable case is the IPSL model. In Phase 5, IPSL-CM5 achieved a solid score of 3.974, but its successor on phase 6, IPSL-CM6, reached a score of 4.643, representing a substantial qualitative leap. This progression demonstrates how targeted software quality practices, coupled with the application of static analysis tools, can lead to measurable gains in the maintainability and robustness of scientific models. A deeper evaluation of the development practices and analysis results for this model is available in , providing insight into how these tools contribute to enhanced code transparency and structure.

In general, Phase 6 models show higher overall scores, indicating an upward trajectory in software quality. Models such as HiRAM (4.317), CMCC-CM2 (4.230), and E3SM (4.118) reflect a more mature approach to model development, likely supported by better development workflows, more formalized coding standards, and increased awareness of reproducibility and sustainability requirements.

Taken together, the observed improvements across CMIP phases suggest that software quality is increasingly being recognized as an integral component of scientific modelling. Static analysis tools not only provide objective metrics for evaluating code quality but also deliver actionable insights for scientific software codebases. As demonstrated by the IPSL case, integrating that tools into model development pipelines can yield tangible benefits and should be more widely encouraged throughout the climate modelling community.

Future climate modelling efforts should incorporate the findings presented in this study. Researchers developing new models from scratch, and therefore not constrained by legacy code, are particularly well positioned to implement appropriate licensing practices, comprehensive documentation, and reproducibility protocols from the outset.

It is advisable that final, production-ready versions of climate models used to support conclusions in international climate assessments, together with their corresponding simulation outputs, be archived in long-term, trusted repositories. This recommendation applies broadly to all model intercomparison initiatives. Given the central role of climate models in assessing climate change, restricted access to source code or configuration details may be perceived as a methodological vulnerability. The inability to reproduce published results due solely to missing code or insufficient documentation represents a significant barrier to scientific transparency. It is also crucial to emphasize the distinction between reproducibility and replicability : reproducibility refers to obtaining identical results under identical conditions, whereas replicability refers to achieving consistent results under varied conditions. Both dimensions are essential for scientific reliability and trust.

The combination of final model releases with modern computational solutions, including cloud infrastructures, containerization technologies, and workflow automation, provides a promising avenue toward achieving full replicability in Earth system modelling . Such approaches have already demonstrated their value in other scientific domains by reducing technical barriers and fostering more robust computational environments.

At the same time, climate model evaluation frameworks continue to evolve. Validation tools employing a wide range of performance metrics are increasingly used to assess model output. In this context, tools such as ESMValTool , which was specifically designed for systematic evaluation, offer an important opportunity: to integrate code accessibility, licensing, and code quality evaluation assessments into the CMIP evaluation workflow. Incorporating these dimensions would encourage more consistent standards across modelling centres and improve transparency.

Scientific gaps identified in CMIP5 further highlight the need for open-source policies. Without access to the underlying source code, it becomes impossible to diagnose discrepancies between models or resolve differences in methodological implementation, ultimately hindering scientific progress. Experience in other areas of software development shows that open code sharing contributes to better implementation practices, improved reliability, and more efficient error detection . Moreover, open collaboration supports the collective scientific effort required to address climate change and aligns with core principles of scientific integrity .

Funding agencies and research institutions therefore have a critical role to play. Allocating resources not only for model development but also for documentation, reproducibility workflows, and long-term code preservation is essential for ensuring that climate modelling keeps pace with evolving expectations around transparency and open science.

Although challenges related to computational scientific reproducibility (CSR) persist across scientific disciplines, these challenges also present significant opportunities. In a field as societally vital as climate research, addressing the structural and organizational barriers identified here will strengthen the robustness, credibility, and long-term sustainability of the climate modelling enterprise.

This study provides a systematic evaluation of source code accessibility and static code quality across all historical phases of the Coupled Model Intercomparison Project (CMIP). Our analysis confirms that Fortran remains the predominant programming language in climate model development, reflecting its long-standing role in scientific computing and its extensive legacy within Earth system modelling frameworks.

A central outcome of this work is the persistent lack of accessibility to the source code of most CMIP models. This limitation continues to hamper reproducibility, a principle explicitly emphasised in current scientific standards and journal policies. The restricted availability of model code arises from several factors, including licensing constraints, dependencies on proprietary components, and the incomplete preservation of early CMIP model versions. Additionally, structural aspects of the research environment – such as production-oriented publication pressures and the low prioritisation of long-term repository maintenance – further exacerbate these issues. Expanding the use of robust, long-term digital preservation platforms and integrating validation tools throughout the development cycle would meaningfully strengthen the transparency of climate modelling.

To enhance reproducibility and replicability, it is essential that final, fully documented versions of climate models be published in stable public repositories. We also recommend incorporating code analysis tools such as FortranAnalyser into CMIP evaluation workflows to improve the traceability, maintainability, and robustness of model implementations. Furthermore, adopting FLOSS (free-libre open-source software) practices and allocating dedicated funding to support collaborative development would facilitate broader community engagement and harmonisation of software standards.

Based on the findings of this work, several priority directions emerge for the CMIP community:

Improving code accessibility: Overcoming licensing barriers, minimising reliance on proprietary dependencies, and promoting deposition of code and documentation in long-term repositories (e.g. Zenodo) are necessary steps. Establishing clear requirements for model availability within international assessments would also support reproducible experimental design.

The challenges identified in this study are technical, organisational, and cultural. Addressing them requires coordinated action across modelling centres, infrastructure providers, and the broader scientific community. Improving code accessibility, software quality, and transparency will enhance the reliability of climate models and reinforce their role as essential tools for understanding and projecting climate change.