The REF was designed to be a community-owned evaluation framework that leverages existing open source, community-built model evaluation and benchmarking packages that are integrated together through a standard application programming interface (API) that will execute modules for generation of diagnostics and the metrics that underlie them. By incorporating existing tools and metrics with publicly available reference datasets, the REF standardizes a community workflow for CMIP activities, reduces duplication of efforts for evaluating models, and, through deployment on the Earth System Grid Federation (ESGF), provides rapid feedback to the research community about relative model performance across a wide range of model components and variables. Moreover, the REF is expected to offer a starting point for the research community to expand model evaluation and benchmarking capabilities and applications within their own institutions or modeling centers.

The high-level workflow for the REF as it runs on ESGF nodes, as shown in Fig. , is triggered by the publication of new simulations requiring evaluation. After the model output has undergone a quality control check to assure the metadata are correct, data storage and compute resources are allocated for a new execution of the diagnostics generation process. As agreed upon by the CMIP Panel, model output that does not conform to the required Controlled Vocabulary (CV) and metadata standards will not be evaluated by the REF. Next, an optimized directed acyclic graph of tasks is produced, and processes are initiated to calculate model evaluation metrics and construct diagnostics. The outputs of diagnostics are then staged on public websites for sharing with the research community. The initial implementation of the REF was created to evaluate Assessment Fast Track simulations , using five to six diagnostics across five Earth system realms with the expectation that additional diagnostics would be added in the future.

Modeling centers usually evaluate their models during development with the overall performance documented and published once the models are finalized and key simulations are completed. However, this approach has three main limitations. First, the process is slow, and evaluation results often come too late to inform assessment reports or even later for stakeholders who need timely information. Second, these evaluation results are hard to access; they are scattered across different types of papers (technical or peer-reviewed, open access or not, etc.) across the modeling centers and often only partially included in IPCC reports, making them difficult to locate and piece together. Third, no consistent approach is applied across centers; each group runs its own evaluations using sometimes different methods, which makes comparisons between models difficult. The REF aims to address these issues by bringing together core evaluation frameworks and their diagnostic output in one place. This workflow is envisioned to help experts quickly assess, understand, and synthesize the performance of a new generation of ESMs for Assessment Fast Track simulations. The REF also makes it easier for stakeholders to access the information they need to support regional analysis for adaptation and mitigation, and it supports best practices for evaluation during model development. Running the REF before model output is submitted to ESGF offers modeling centers an additional option to systematically assess their models and to make targeted improvements during the development phase.

The REF aims to integrate well-documented reference datasets for comparison with model output. These reference datasets will be collected in obs4MIPs , a project created to distribute data that support evaluation of ESMs via ESGF (nodes listed at https://esgf.github.io/nodes.html, last access: 3 August 2026). Obs4MIPs compiles a range of observationally constrained data, formatted according to CMIP conventions to ensure compatibility with model output. The published obs4MIPs datasets include variables such as temperature, precipitation, and sea level pressure, spanning various spatial and temporal ranges and scales across Earth system realms (Ocean & Sea Ice, Land & Land Ice, Atmosphere, Earth System, and Impacts & Adaptation), providing a comprehensive basis for model evaluation . The formatting of obs4MIPs data adheres to the Climate and Forecast (CF) conventions and the CMIP metadata standards . These conventions ensure that data are consistently structured, with uniform variable naming, units, and metadata, facilitating easy comparison across model outputs.

While obs4MIPs serves an important role in distributing observational datasets, the REF is flexible and designed to use additional reference data available in the required format. This adaptability allows for the incorporation of new reference datasets as they become available, or when specific needs arise, ensuring that the framework can accommodate a broad range of observational data. The REF leverages the attributes provided by datasets to automate the diagnostic generation process. This automation requires standardization and coherent attributes across models, variables and experiments in order to correctly combine different datasets without manual selection by humans. Reference data used by the REF may span different time periods, a key point that analysts need to consider when interpreting the results of the available diagnostics. Moreover, community activities like CMIP should strive to ensure that reference data span the full contemporary time period of simulation experiments intended to reproduce recent historical trends and variability. The REF does not aim to reformat reference or model data, and REF users must ensure the data they wish to use follow the CF conventions and meet the CMIP metadata standards to be considered REF-compliant data (Fig. ). Quality checks are required to assure the data used for evaluation and benchmarking diagnostics are consistent and reliable, enabling meaningful model-data comparisons . These quality checks do, in some cases, go beyond the checks required for data publication to ESGF.

Key stakeholders for the co-development of the REF were identified as modeling centers involved in the Assessment Fast Track, reference dataset providers, evaluation tool and package developers, diagnostic developers and climate scientists, including IPCC authors, seeking to analyze the Assessment Fast Track output. The MB-TT initiated work on the REF based on the results of the CMIP6 Community Survey , conducted by the World Climate Research Programme (WCRP) from January to March 2022. Detailed analysis of the survey results by the Task Team revealed calls for open and transparent model benchmarking and evaluation and the ability to run CMIP evaluation tools alongside ESGF, as part of the data publication process, with provision of evaluation results display.

Members of Fresh Eyes on CMIP worked with the CMIP-International Program Office and the MB-TT to develop a survey that was circulated to modeling center science and technical leads, as well as more widely in the Fresh Eyes on CMIP and CMIP community mailing lists and via social media. The survey was open 7–17 May 2024, and generated 152 unique responses, and the responses were analyzed by members of the Fresh Eyes on CMIP and reported at a MB-TT workshop. These survey results informed design and development of the initial REF proposal. Suggestions included considering multiple software tools, strategies to ensure accessibility, tools for quality control, and transparency and provenance for reference datasets and model output. Additional suggestions included acceptance of data not processed by the Computer Model Output Rewriter CMOR;, potentially with AI-assisted reformatting, improved accessibility through use of ESGF and cloud-based computing to enhance data inclusivity and flexibility, as well as errata tracking. A detailed analysis of the survey results is available in the survey report by .

A separate community survey was subsequently disseminated to modeling centers in June 2024 that included requests for input on evaluation tools used, interest in use of a benchmarking framework within their own computing environments, and willingness to submit preliminary model output for quality assurance and control checks . This survey informed refinement of the framework structure and implementation plan. In September 2024, a survey was disseminated to modelers, modeling centers, and reference dataset and data infrastructure providers, requesting input regarding the specific diagnostics that should be prioritized for inclusion in the REF, receiving 53 responses. A co-development session with members of the Earth observation community was held at the European Space Agency (ESA) Climate Change Initiative (CCI) and Climate Modelling User Group (CMUG) integration meeting in October 2024, which resulted in the inclusion of ozone-related metrics in existing diagnostics where appropriate and two additional cloud diagnostics. The outcomes of the survey, in combination with suggestions from the MB-TT, resulted in a set of diagnostics, shown in Sect.  and published separately . The REF was thereby launched on 4 November 2024, at a community-engagement event that saw wide participation. Successive months were dedicated to implementation of the diagnostics from four existing evaluation and benchmarking packages within the REF architecture. The packages selected for providing diagnostics for the Assessment Fast Track REF were ESMValTool, PMP, ILAMB, and IOMB, which are described below in Sect. . During this period, engagement efforts focused on technical implementation, involving members of the community responsible for provision of supporting infrastructure and quality control, including the WGCM Infrastructure Panel (WIP), obs4MIPs Steering Panel, the ESGF-WIP Quality Assurance and Control Working Group, the CMIP Data Request and CVs Task Teams, as well as the IPCC Task Group on Data Support for Climate Change Assessments (TG-Data), on their citation and provenance requirements for the REF, see Sect. .

A preliminary prototype of the working suite was stress-tested for operational usage in time for the REF Hackathon, which was held at the Met Office, in Exeter, UK, from 10–13 March 2025, and included dedicated drop-in sessions for modelers and reference dataset providers. This prototype was able to ingest a sample of CMIP6 model output and obs4MIPs data, running a small subset of ILAMB, PMP, and ESMValTool diagnostics. In May 2025, modeling center science and technical personnel, as well as previous attendees of the REF launch, were invited to attend the launch of the beta version of the REF. Approximately 107 participants registered for the demo and to attend follow-up feedback sessions in June 2025 regarding the beta release. This beta testing period was instrumental for building an easy-to-use tutorial and documentation, solving potential issues with REF usage on HPC machines, and receiving feedback from users, with the aim of delivering a public release of the REF for actual usage with Assessment Fast Track outputs in October 2025.

The open source evaluation and benchmarking packages described below – ESMValTool, PMP, and ILAMB & IOMB – were chosen for inclusion in the first version of the REF because they are open source, were relatively mature packages at the time, and combined, they permit comprehensive model evaluation across all the atmosphere, ocean and sea ice, land and land ice, Earth system, and impacts and adaptation realms. Subsets of diagnostics from each package were selected based on input from the MB-TT and the community. For the initial version of the REF, diagnostics were restricted to analyze only monthly mean output from the expected new simulations. In this section, the packages are briefly described, although a more in-depth description of them can be found in , and the Coordinated Model Evaluation Capabilities (CMEC) standards are also described, since they offer a strategy for integrating these disparate evaluation packages.

At the heart of the REF are the diagnostics that were selected in an iterative process (see Sect. ) that can be calculated with each new simulation that is presented to the REF. The diagnostics are grouped according to the five different realms that were identified for the CMIP data request and each diagnostic included in this first version of the REF is calculated by only one software package. Table  provides an overview of all the diagnostics that were selected for the first version of the REF, based on the diagnostics table published on Zenodo , their realm, the software package with which the diagnostic is calculated, and the reference datasets used in the comparison. A more detailed table of the diagnostics is presented in Appendix . For some of the diagnostics, different methodologies adopting different software packages across the diagnostic providers were discovered (e.g., double Inter-tropical Convergence Zone (ITCZ) biases); in this case, the principle of minimal computational resources and least number of required variables was adopted in order to choose the appropriate tool to be responsible for the diagnostic in the REF.

During the REF development phase, it became clear that two of the identified diagnostics were not available from the software packages implemented in the Assessment Fast Track REF. These diagnostics, with their unique IDs, 5.1 (High amplitude Rossby waves) and 5.2 (Internal variability or ensemble spread for individual models), were then removed from the list of those to be integrated in the initial REF version, following consultation and agreement from the Impacts & Adaptation CMIP Data Request Author team leads and co-chairs of the Vulnerability, Impacts, Adaptation and Climate Services (VIACS) Advisory Board. One additional diagnostic was removed from the diagnostic list, ID 1.5 (Ocean heat content (OHC)), since also for this diagnostic no code for its calculation was available from the software packages. The original diagnostics ID 3.4 (Evaporation minus precipitation (E-P)) was moved to a different realm (now ID 2.8 (Precipitation minus evaporation (P−E))), and diagnostic ID 3.5 (Double inter-tropical convergence zone (ITCZ)) was added to another existing diagnostics (ID 1.3 (El Niño Southern Oscillation (ENSO) diagnostics (lifecycle, seasonality, amplitude, teleconnections))) to better reflect the actual calculated metrics.

A more detailed description of each diagnostic and the rationale and methodology for its implementation is presented in Appendix .

For CMIP7, the data request to modeling centers is based on different “opportunities” that the community could submit to the CMIP IPO, following a community-wide call, each focusing on a specific scientific topic. Each opportunity contains a concise description of its scientific goals and a list of all model variables that are requested from simulation output to achieve these goals. The different data request opportunities are grouped according to five realms (Ocean & Sea Ice, Land & Land Ice, Atmosphere, Earth System, and Impacts & Adaptation), which are the same as those that were used as a basis for selecting REF diagnostics (see Sect. ).

An opportunity regarding the REF was submitted to the Data Request Task Team's open call to ensure that the model variables required by the REF for generating the REF diagnostics were clearly identified and that modeling centers wishing to use the REF with their simulations would have a checklist of model variables needed for producing the REF diagnostics. While some REF diagnostics use only a single model variable from the historical simulation, others require two or more model variables from multiple different simulations, as described in the definitions in Sect. . Since the REF diagnostics span all five data request realms, a variable group request was submitted for each realm. The variable groups combined form the “REF CMIP7 data request” opportunity. The opportunity was published as part of the v1.2.1 release of the data request, consisting of five variable groups, containing 80 variables in total. Many of the variables are from the recently developed list of baseline climate variables for Earth system modeling , which is a subset of variables reflecting the most frequently used elements of CMIP6. To facilitate finding information about the REF opportunity in the different data request documentation papers, it was decided that each paper would contain a very brief description of the opportunity and refer to the documentation paper about the atmosphere realm, where a more detailed REF opportunity description was added .

Observations play a key role in the REF and several diagnostics require observed quantities for different climate variables across domains for model evaluation. For each diagnostic, we have identified at least one reference dataset for inclusion in the REF (see complete list in Appendix ). To ensure compliance with the Findability, Accessibility, Interoperability, and Reusability (FAIR) data standards , all observational datasets included in the REF are required to have a fully open access license (e.g., CC-BY-4.0, CC0, OGL) and follow the CF conventions to ensure technical alignment with CMIP standard output. This is enabled by making the datasets available for downloading on ESGF servers as part of the obs4MIPs project . For any datasets not meeting the open access criteria, the REF Delivery Team obtained a relaxation of the license constraint by formally requesting and receiving agreement from individual dataset providers. Observational datasets for the REF were then processed in compliance with the obs4MIPs Data Specifications 2.5 (ODS2.5) with approval from the obs4MIPs Steering Panel (OSP). In facilitating observational data ingestion for the REF, two additional criteria not fully addressed by the ODS2.5 specifications were identified, for which the REF delivery team has drafted guidelines. These criteria relate to the treatment of uncertainty information in the data and for the formulation of complex citations (see discussion in Sect.  and ). In an effort to recognize the need for observational record lengths to be as closely aligned with CMIP simulations as possible, we specify that observational datasets should be available at least until the end of 2016, ensuring that the data is less than five years out of date with historical simulations from CMIP7, that run until the end of 2021. We consider the availability of data towards the end of the historical simulation period to be important for the calculation of certain statistical properties such as trends used in evaluation. Where the observational datasets currently do not meet this requirement, we have highlighted such entries in Table  in Appendix A, so that the data sets may be extended or replaced as appropriate in the future.

Two pathways were identified for observational dataset providers to submit their datasets for future inclusion in the REF. The first step in both cases is to submit a dataset proposal to obs4MIPs for approval by the OSP, which can either be done directly by the dataset provider or via a third party with permission from the dataset provider. Once the dataset has been approved by the OSP, the registered content, including the dataset name, version, data provider details and release date should be submitted to the Program for Climate Model Diagnosis and Intercomparison (PCMDI) obs4MIPs CMOR Tables repository (https://github.com/PCMDI/obs4MIPs-cmor-tables, last access: 3 August 2026). This process provides the dataset with a unique source_id, following CMIP conventions. Datasets can then be prepared for compliance in one of two ways:

using the CMOR software as advised by the OSP to prepare their dataset, following instructions and examples on the PCMDI obs4MIPs CMOR GitHub repository (the “CMOR pathway”).

The CMOR pathway intrinsically provides a form of validation for datasets before publication through the use of the CMOR software. As the CMOR-like pathway may not provide the same level of validation, the REF Delivery Team has developed a validation script that the prepared CMOR-like datasets must pass before publication. The prepared datasets are then published to ESGF via one of the two Assessment Fast Track REF nodes, at the Center for Environmental Data Analysis (CEDA, United Kingdom) or at Oak Ridge National Laboratory (ORNL, United States of America).

For certain diagnostics (1.1 and 5.4 in Table ), the reference datasets listed in Table  are pre-processed to produce one or more static files with information needed for the diagnostic. In such instances, these files are stored internally within the corresponding diagnostic package, and the REF includes acknowledgments and references for the input datasets used to produce the files. For such exceptions, the input datasets themselves are not required to be published on ESGF.

Currently, most diagnostics within the REF do not incorporate uncertainty information from reference datasets. Where available for a reference dataset used in the diagnostics, the standard error of the mean or upper and lower bounds around the mean are used. Future plans for the REF include using multiple reference datasets to account for observational uncertainty as seen commonly in the literature. The unavailability of comprehensive and accurately characterized uncertainty information with the observational data was identified as a key barrier to incorporating this information by diagnostic developers. Feedback from observational data providers indicated that any associated uncertainty and instructions on how to correctly apply such information is best provided by the data providers themselves. This prompted the REF Delivery Team to develop guidance for including uncertainty information in observational datasets; which was necessary to allow ingestion of uncertainty information by the REF. The initial proposal, originating from community engagement at the REF Hackathon with observational dataset providers and metrics package developers, was refined following consultation with the OSP and the CMIP-CVs Task Team. For the CMOR pathway, the following requirements are outlined:

All additional uncertainty information (see Table  for currently accepted information on uncertainty) is provided in a separate file. Initially, between one and three additional files will be used, depending on the extent of uncertainty information provided.

This distinction between how uncertainty information may be included in CMOR and CMOR-like datasets is due to the current capabilities of the CMOR software. Currently, CMOR does not accommodate the addition of ancillary variables in its output, and the required software update for inclusion is not feasible within the Assessment Fast Track REF timeline. For the CMOR-like pathway, there is no requirement to add the uncertainty information in separate files. Instead, additional uncertainty fields may be provided following CF conventions, by adding them as ancillary variables in a netCDF-CF file. The additional global attributes has_aux_unc and aux_uncertainty_id are still required, and the variable_id should be constructed as described above only using the suffixes from Table . This guidance was developed as a basic framework to enable the community to utilize a wider range of uncertainty information during the development of REF diagnostics.

The REF Delivery Team consulted with the IPCC Task Group on Data Support for Climate Change Assessments (TG-Data) on their citation and provenance requirements for the REF. The IPCC TG-Data aims to enhance the traceability of key results presented in the AR7 AR7; cycle using the new Complex Citation standard for documentation and standardized provenance records for gathering the required information. This simple but flexible Complex Citation approach allows for the traceability of data product generation and the citation of multiple datasets or data subsets in a single referenced object called Complex Citation Object (CCO). TG-Data aims to include CCO references in every figure caption of the AR7. Prerequisite to this is the provision of detailed information on input data usage in the form of persistent identifiers (PID) for each file and each citable entity.

The REF has identified the need for the reference datasets to be published on ESGF with their Handle IDs and data collections with their DOIs. ODS2.5 already requires each file to have a global attribute called tracking_id used as PIDs on ESGF, a unique identifier with prefixes specified for each ESGF project. In addition, reference datasets used by the REF also require the inclusion of the DOI as a global attribute to ensure compatibility with CCO. The REF leads the task of defining a provenance template for authors of the IPCC AR7 as guidance on providing the CCO-related information.

Through the CMIP7 data citation, the CCO captures provenance information at the granularity of a model's contribution to an experiment. The list of the file handles of the specific data that were used from within the CMIP7 citation resource provides traceability.

The REF workflow consists of four steps:

Ingestion: The user registers the source datasets that can be used (reference data and model output). The associated metadata is extracted and added to a local database along with the respective file paths. Only ingested files are used in any execution calculations.

Figure  shows how the key services for the REF are integrated within the ESGF deployment. At the core of this system is the Compute Engine, which orchestrates the workflow by managing data ingestion and processing tasks. The Compute Engine manages a database (PostgreSQL or SQLite) of all the datasets, executions and results that have been performed. The Compute Engine will ingest new data when they become available either via consuming newly replicated events from the ESGF-Next Generation Event Stream, a Kafka-based service, or by periodically checking for newly available datasets. A “solve” is then be performed to determine if any new executions are required. For the ESGF deployment, each of the services (blue boxes in Fig. ) are deployed as a Kubernetes service.

Data processing is handled by the Compute Engine Worker, which pulls executions from a queue and then executes the appropriate diagnostic. Each diagnostic provider is deployed as a separate service that can be scaled independently according to demand. This might be particularly important if one of the execution services requires more computational resources than the others. These workers require access to shared read-only mount containing input data (CMIP and observation datasets) and a mount for the Scratch Datastore where results are stored.

The Compute Engine watches for the success/failure of the executions on the workers. A subset of results are copied to the Output Datastore by the Compute Engine. During this process, metadata about the results are extracted into the database, including metric values, figures and data files. The use of a central database of results allows for efficient querying in the API across executions. This is critical to enable the ability to make dynamic data requests in the API. Users can access the REF through two primary interfaces: by directly querying the Data API, and the Portal, which consumes the Data API and serves as the landing page for information about the REF and provides synthesized results.

The REF can be deployed without the need for Kubernetes or docker depending on the constraints of the deployment target. Non-ESGF deployments will look very similar except that the ingestion and solving is performed via the Command Line Interface (CLI). The CLI is used to manually interact with the REF and its database, and is how users interact with the REF in a small deployment.

The REF operates on CF-compliant netCDF model output and reference datasets aligned with CMIP CV. When run within ESGF deployments, the REF is not be required to evaluate simulations that fail to meet required data-quality standards; if simulations cannot be processed by the REF, these simulations will be flagged or excluded from the public dashboard. The REF automatically accesses published model output and reference data; however, the REF also supports the analysis of datasets that are not published on ESGF, which allows modeling centers and individual researchers to perform their own model evaluations. A deliberate decision was made to enable the evaluation of datasets and model output that do not conform with the CMIP CV, enabling assessment of model versions not intended to be published on ESGF. In such offline cases, the local datasets must be available to all of the execution services, and ideally should have undergone a quality control (QC) process to assure that the metadata are correct and the data represent the desire output in the correct units.

The REF is designed to run on a variety of hardware platforms, from desktop computers to high performance computing (HPC) environments, and can be run either within virtual Conda environments or within Docker containers. Depending on the use case, either may be used; however, Docker containers are recommended for a production deployment, as they can be scaled independently. The instructions for installation and getting started are available via the REF documentation site at https://climate-ref.readthedocs.io/en/latest/ (last access: 12 June 2026) or the GitHub repository at https://github.com/Climate-REF/climate-ref (last access: 12 June 2026) .

The REF Delivery Team was composed of an international team that brought together three different evaluation package providers, each of which possessed their own workflows, assumptions, and conventions. An independent delivery team leader was chosen to coordinate the development and architect the end-to-end workflow. Significant time was invested in identifying commonalities among the providers with the goal of minimizing the amount of additional development needed to integrate all the selected diagnostics into the REF. These tools were never designed to be interoperable and have long-standing communities that use the packages outside of the REF, so a requirement for minimal changes to the underlying packages was also mandated. Developing the appropriate abstractions and common language within the REF depended upon ongoing discussions and often required the prototyping of multiple potential solutions.

The REF system can be quite complex since it is composed of multiple services. A key requirement for the framework was to make it easy for additional packages to be integrated into the REF, both now and in the future. Care was taken to ensure that the amount of context that must be understood for a package developer to contribute diagnostics was kept to a minimum. The decision for the structure of the framework has two impacts, it provides a separation of concerns and minimizes the complexity required simply to contribute diagnostics, as well as providing the scope to support the different assumptions made by different providers. The ability to quickly refactor and modify the interfaces between the packages was critical. The use of a monolithic repository, where all of the REF-related packages were in a common GitHub repository with issue tracking and a Kanban board, was crucial for development and testing. Additionally this structure was important to enforce the need for type hints throughout the code base and providing high test coverage. The REF is openly developed and available on GitHub at https://github.com/Climate-REF/climate-ref under the Apache License 2.0 open source license. The version of the REF contemporaneous with the submission of this manuscript is archived on Zenodo at https://doi.org/10.5281/zenodo.15103441 .

The REF Hackathon, hosted by the UK Met Office in Exeter in March 2025, was convened to bring together the REF Delivery Team and to solicit early feedback from potential users. The Hackathon successfully accelerated code development and reference data conversion, obtained feedback and facilitated discussions about approaches for computing diagnostics and supporting traceability for CCO, and tested an alpha version on the UK Met Office HPC facilities. The Hackathon was held relatively early in the project development lifecycle and significant time was spent workshopping the conceptual framework of the REF. After the Hackathon, the research community continued to offer ideas for additional capabilities that could be enabled by the REF framework. A CMIP6-ready version of the REF was subsequently launched at the ESM2025 Final General Assembly on 7 October 2025. It was then further refined based on user feedback and the CMIP7-read version was launched at the CMIP 2026 Community Workshop held in March 2026 in Kyoto, Japan.

The CMIP Panel initiated the MB-TT, with a call for members in November 2022. The intention of the Task Team was to put together a group of experts who would work together to provide a systematic, open, and rapid performance assessment of the expected large number of models participating in CMIP7, providing a set of informative diagnostics and performance metrics. The Task Team originally focused on assessing evaluation approaches and software packages. However, it then became apparent that there was a larger opportunity here to fully integrate evaluation tools into the CMIP publication workflow with diagnostic outputs published alongside model output on ESGF and results displayed on an easily accessible website.

The REF became an extremely attractive option to the CMIP panel as it enables (1) consistent assessment of CMIP output by modeling centers, (2) support for author teams in the context of the IPCC and other national climate assessments and (3) input for model selection for downstream applications. The modeling centers are essential to the CMIP endeavor, so having a tool that supports their activities is the most important factor. Many of the CMIP panel members have contributed to IPCC assessments or national climate assessments, so it was clear that there was an opportunity to support such reports if model diagnostics were easily available to authors (reducing the workload of both authors and chapter scientists). Wider community discussions facilitated by the CMIP panel have also revealed enthusiasm from users of CMIP output for regional downscaling and impacts applications. The REF allows to have the diagnostics quickly available to support the choice of a limited selection of climate model output to serve their particular application.

The CMIP Panel view the REF as a potential game changer for users of CMIP data. CMIP Panel co-chair Helene Hewitt reflected, during opening remarks at the March 2025 Hackathon on-boarding session, that “as someone that was a Coordinating Lead Author of the IPCC AR6 WGI, I saw how much work our chapter scientists did in producing all the metrics and plots, this builds on the then-CMIP6 Panel vision, we are so happy that the Task Team has taken this on. We hope that being able to integrate these tools into the workflow and ESGF publication process, on a website, will make it much easier for users to know what they are looking at. Going forward we want to streamline this chain, supporting model selection and downstream use of CMIP.” Following the launch of the REF in March at the CMIP 2026 Community Workshop, held in Kyoto, Japan, co-chair of the CMIP Panel John Dunne said, “[T]his release marks an important milestone advancing global accessibility and usability of CMIP data.”

The REF will produce diagnostics spanning a variety of metrics and figures, as well as calculate scalar scores as a gauge of correspondence between model output and reference datasets. The scores are not meant to discriminate “good models” from “bad models.” Instead, the REF is intended to assist analysts in quickly identifying relative differences among models or model versions that researchers must then interpret by viewing the plots and maps that underlie the individual diagnostics. REF users must consider that only a limited number of diagnostics can be produced and evaluated and that reference datasets bring with them their own (usually unquantified) uncertainties. While it is hypothetically possible that modelers could “tune” or “overtune” their models to score well on a small set of specific diagnostics, this approach to improving model performance in the REF is of limited practical utility since the reference data are not consistent with each other and the relatively large number of diagnostics and metrics makes such an optimization impossible for a physics-constrained model. We emphasize that while the REF provides a suite of valuable diagnostics of model performance, it cannot replace dedicated analyses for each diagnostic that investigate in detail the mechanisms behind different model behaviors. Model improvements must come from such mechanistic understanding, and the REF may be useful in highlighting where to conduct such a detailed investigation.

Most multi-model assessments indicate that each model has strengths and weaknesses in different areas. For example, one model may perform well with regard to the distribution of precipitation but may not exhibit the desired distribution of sea surface temperatures, while other models may show the opposite behavior. Similarly, some models may capture biogeochemical responses on land well but have a weak representation of the hydrological cycle, while other models may capture runoff well but fail to capture the seasonal cycle of terrestrial productivity. The REF results are best used to identify which subset of models may be best for individual studies based on their performance in key science areas or spatial regions of interest. REF results may help inform the selection of models to include in downscaling studies or the choice of multi-model weights when optimizing for a particular metric. Of course, analysts must consider a wide range of factors when making such selections, including performance on a variety of metrics for relevant model variables (across spatial regions and through time) that may go beyond those incorporated into the initial version of the REF. All of these suggested uses require the researcher to “drill down” into the detailed results produced in the diagnostics, looking beyond performance scores that are merely a high-level indicator of large-scale average correspondence of model output with a reference dataset.

To demonstrate the utility of the REF, some example use cases, based on CMIP6 simulations, are described below for the atmosphere, land, and ocean components of models. These few simple examples are intended to highlight potential applications of the REF, not to report any of the features of models named in the descriptions below.