The process of defining numerical experiments is potentially complex (Fig. a). It begins with an idea and often entails an iterative community discussion which results in the final experimental definition and documentation. In the simplest cases, such documentation may be prose, in a paper or a journal article, but when many detailed requirements are in play and/or many experiments and individuals are involved, it is helpful to structure the documentation – both to ensure that key steps are recorded and to aid in the inter-comparison of methodology between experiments (especially the automatic generation of tables and views). Key requirements include being very specific about imposed experimental conditions and the required output.

Once the experiments are defined (Fig. a), modelling groups realize the experiments in the form of simulations which attempt to conform to the specifications of the experiment and which produce the desired output (Fig. b).

In both generic experiment documentation and in defining data requests, it is helpful to utilize controlled vocabularies so that unambiguous machine navigable links can exist between the design documentation, simulation execution, data production, and the analysis outcomes.

The requisite controlled vocabulary for a numerical simulation workflow requires addressing the actions and artefacts of the workflow summarized in Fig. , in which we see Projects (e.g. MIPs) design NumericalExperiments and define their NumericalRequirements. (In this section we use italics to denote specific concepts in the ES-DOC taxonomy.) Experiment definitions are adopted by modelling groups who use a model to run Simulations, with Output Data requirements (“data requests”) being one of the many experimental requirements. A simulation is run with a Configured Model, using a configuration which will include details of InputData and may include Modifications required to conform to the experiment requirements. Not all of the configuration will be related to the experiment, aspects of the workflow and computing environment may also need to be configured. In practice, simulations can deviate in detail from the experiment protocol; that is, they do not conform exactly to the requirements. A key part of a simulation description, then, is the set of Conformance descriptions which indicate how the simulation conforms to the experimental requirements. In this paper we are limiting our attention to the definition of the Experiment and its Requirements, with application to CMIP6 and the relationship between the MIPs and those requirements. We address other parts of the workflow elsewhere.

As noted above, a project has certain scientific objectives that lead it to define one or more NumericalExperiments. We describe the rules for performing the numerical experiments as NumericalRequirements (Fig. ). Both NumericalExperiments and NumericalRequirements may be nested and the former may also explicitly identify specific related experiments which may provide dependencies or other scientific context such as heritage. For example, an experiment from which initialization fields are obtained is referred to as the parent experiment. Nested requirements are used to bundle requirements together for easy reuse across experiments. (An example of a nested requirement can be seen in the Appendix, where Table  shows how all the components which go into a common CMIP6 pre-industrial solar particle forcing are bundled together. We will see later that in CMIP6, many implicit relationships arise from common requirements.)

The experiment description itself includes attributes covering the scientific objective and the experiment rationale addressing the following questions: what is this experiment for and why is it being done?

The NumericalRequirements are the set of instructions required to configure a model and provide prescribed input needed to execute a simulation that conforms to a NumericalExperiment. These instructions include (Fig. ) specifications such as the start date, simulation period, ensemble size, and structure (if required), any forcings (e.g. external boundary conditions such as the requirement to impose a 1 % increase in carbon dioxide over 100 years), initialization requirements (e.g. whether the model should be “spun-up” or initialized from the output of a simulation from another experiment), and domain requirements (for limited area models). A scope keyword from a controlled vocabulary can be used to indicate whether the requirement is reused elsewhere, e.g. in the specifications for related experiments.

Each requirement carries a number of optional attributes and may contain mandatory attributes, as shown in Tables  and  for a ForcingConstraint.

The ES-DOC vocabulary is an evolution of the “Metafor” system , developed to support the fifth CMIP phase (CMIP5). Metafor was intended to provide the structured vocabulary and tools to allow those contributing simulations to CMIP5 to document their models and simulations. In that context, Metafor was a qualified success; useful information was collected, but the tools were not able to be fully tested before use and were found to be difficult to use by those providing the documentation content. Such difficulties resulted in documentation generally arriving too late to be of use to the target audience: scientists analysing the data. The lessons learnt from that exercise were baked into the ES-DOC project which has superseded Metafor, leading to a much improved ontology, better tooling, and improved viewing of the resulting documentation (https://es-doc.org, last access: 2 March 2020).

The ES-DOC controlled vocabulary is an instance of an ontology (“a formal specification of a shared conceptualization”, ). There is considerable literature outlining the importance of such ontologies in establishing common workflow patterns with the goal of improving reproduction of results and reuse of techniques (whether they be traditional laboratory experiments or in silico) and explicitly calling out the failure of published papers as a medium to provide all the details of experiment requirements (e.g. , in the context of reproducible machine learning).

The description of ontologies is often presented in the context of establishing provenance for specific workflows and often only retrospectively. Work supporting scientific workflows has mainly been concerned with execution and analysis phases, with little attention paid to the composition phase of workflows , let alone the more abstract goals.

For the “conception phase” of workflow design, a controlled vocabulary introduced by as part of their proposed description of “experiment life cycles” directly maps to our work on experiment descriptions (discussed in this paper). In their view, the conception phase potentially consists of an abstract workflow, describing what should be done (but without specifying how), and a concrete workflow, binding abstract workflows to specific resources (models, algorithms, platforms, etc). ES-DOC respects that split with an explicit separation of design (experiment descriptions) and simulation (the act of using a configured model in an attempt to produce data conforming to the constraints of an experiment).

The notion of “an experiment” also needs attention, since the experiments described here are even more abstract than the notion of “a workflow” and cover a wider scope than that often attributed to an experiment. Dictionary definitions of “scientific experiment” generally emphasize the relationship between hypothesis and experiment (e.g. “An experiment is a procedure carried out to support, refute, or validate a hypothesis. Experiments provide insight into cause-and-effect by demonstrating what outcome occurs when a particular factor is manipulated.”, ). In this context “factor” has a special meaning, a factor generally being one of a few input variables; but in numerical modelling there can be a multiplicity of such factors, leading to difficulties in formal experimental definition and consistency of results ( in the context of big data experiments).

The first formal attempt to define a generic ontology of experiments (as opposed to workflows), appears to be that of (who also expressly identify the limitations of natural language alone for precision and disambiguation). Key components of their ontology include the notions of experimental classification, design, results, and their relationships, but it is not obvious how this ontology can be used to guide either conception or implementation. To specify more fully the abstract conception phase of workflow with more generic experiment concepts, build on with much the same aim as ; however, they introduce many elements in common with ES-DOC, and one could imagine some future mapping between these ontologies (although there is not yet any clear use case for this).

With the advent of simulation, another type of experiment (beyond those defined earlier) is possible: the simulation (and analysis) of events which cannot be measured empirically, such as predictions of the state of a system influenced by factors which cannot be replicated (or which may be hypothetical, such as the climate on a planet with no continents). For climate science, the most important of these is of course the future; experiments can be used to predict possible futures (scenarios).

In this form of experiment, ES-DOC implicitly defines two classes of “controllable factor”: those controlled by the experiment design (and defined in NumericalRequirements, in particular, by constraints) and those which are controlled by experiment implementation (the actual modelling system). Only the former are discussed here. Possibly because most of the existing work does not directly address this class of experiment, there is no similar clear split along these lines in the literature we have seen.

Global model inter-comparison projects have a long history, with pioneering efforts beginning in the late 1980s (e.g. ). The first phase of CMIP was initiated in the mid 1990s . CMIP1 involved only a handful of modelling groups, but participation grew with each succeeding phase of CMIP. Phase six (CMIP6, underway now) will involve dozens of institutions, including all the major climate modelling centres and many smaller modelling groups. Throughout the CMIP history, there has been a heavy reliance on CMIP results in the preparation of Intergovernmental Panel on Climate Change (IPCC) reports – CMIP1 diagnostics were linked to IPCC diagnostics and the timing of CMIP phases has been associated with the IPCC timelines.

With each phase, more complexity has been introduced. CMIP1 had four relatively simple goals: to investigate differences in the models’ response to increasing atmospheric CO2, to document mean model climate errors, to assess the ability of models to simulate variability, and to assess flux adjustment . CMIP6 continues to address the first three of these objectives (flux adjustment being rarely used in modern models), but with a broader emphasis on past, present, and future climate in a variety of contexts covering process understanding, suitability for impacts and adaptation, and climate change mitigation.

In CMIP5 and again in CMIP6, there was a substantial increase in the number and scope of experiments. This has led to a new organizational framework in CMIP6 involving the distributed management of a collection of quasi-independently constructed model inter-comparison projects, which were required to meet requirements and expectations set by the overall coordinating body (the CMIP Panel) before they were “endorsed” as part of CMIP6. These MIPs were designed in the context of both increasing scope and wider-spread interest and the growth of two important constituencies: (1) those designing “diagnostic MIPs”, which do not require new experiments, but rather request specific output from existing planned experiments to address specific interests, and (2) the even wider group of downstream users who use the CMIP data opportunistically, having little or no direct contact with either the MIP designers or the modelling groups who ran the experiments.

With the increasing complexity, size, and scope of CMIP came a requirement to improve the documentation of the activity, from experiment specification to data output. CMIP5 addressed this in three ways: by documenting the experiment design in a detailed specification paper ; by improving documentation of metadata requirements and data layout to improve access to, and interpretation of, simulation output; and by requiring model participants to exploit the Metafor system (Sect. ) to describe their models and simulations. ES-DOC use is now required for the documentation of CMIP6 models and ensembles .

The overview of the experiment design process given in Fig.  can be directly applied to the way many of the CMIP6 MIPs were designed. For CMIP6 the iterative process involved the CMIP Panel,

https://www.wcrp-climate.org/wgcm-cmip/cmip-panel, last access: 2 March 2020

Somewhat naively, the initial concepts for ForcingConstraints anticipated the description of forcing in terms of specific input boundary conditions or, perhaps, specific modifications needed to models – this was how they were described for the CMIP5 documentation. The ES-DOC semantics introduced for CMIP6 are more inclusive and allowed a wider range of possible forcing constraints. For example, in CMIP5 the infamous Metafor questionnaire asked modellers to describe how they implemented solar forcing. In CMIP6, the approach to solar forcing requirements was outlined in the literature , and the resulting requirements are found in rather more precise forcing constraints (with additional related requirements), an example of which appears in Table . The ES-DOC documentation now provides a checklist of important requirements and a route to the literature for both those implementing the experiments and those interpreting their results. Modellers can now use this information both in setting up their simulations and in documenting that setup. A discussion of how the latter is done for CMIP6 (and how it builds on lessons learnt from the generally poor experiences interacting with an excessively long and complicated CMIP5 questionnaire) will appear elsewhere.

Increasing precision is evident throughout CMIP6 and in the documentation. In some cases, rather than ask how it is done in a model post fact, the experiment definition describes what is expected, as in the GeoMIP experiment G7SST1-cirrus (Table ) where explicit modelling instructions are provided. However, where appropriate, experiments still leave it open to modelling groups to choose their own methods of implementing constraints, e.g. the reduction in aerosol forcing described in GeoMIP experiment G6sulfur (Table ).

At the heart of the current CMIP process is a central suite of experiments known as the DECK (Diagnosis, Evaluation, and Characterization of Klima; ). The DECK includes a pre-industrial control under 1850 conditions, an atmosphere-only AMIP simulation with imposed historical sea surface temperatures, and two idealized CO2 forcing experiments where in one CO2 is increased by 1 % per year until reaching 4 times the original concentration, while in the other CO2 is abruptly increased to 4 times the original concentration. Variants of most of these fundamental experiments have been core to CMIP since the beginning, and now within the DECK there is a second variant of the pre-industrial control designed to test the relatively new earth system models which respond to internally calculated CO2 concentrations as opposed to responding to externally imposed CO2 concentration (Table ). Completion of the suite of DECK experiments is intended to serve as an entry card for model participation in the CMIP exercise. The CMIP panel are responsible for the DECK design and definition, which should evolve only slowly over future phases of CMIP and will enable cross-generational model comparisons. CMIP is also responsible for the “historical” experiments, but the definition of these will change as better forcing data become available and as the historical period extends forward in time.

Table  provides a summary of most of the CMIP6 endorsed MIPs as of December 2018, with the DECK incorporated in CMIP as discussed above. This table was auto-generated from the ES-DOC experiment repository (see the Code availability section). It does not include the MIPs that have not originated any of the CMIP6 experiments. There are three of these, which focus on and use CMIP6 output for various purposes: the Coordinated Regional Climate Downscaling Experiment (CORDEX; ) or the three diagnostic MIPs – DYnVarMIP (Dynamics and Variability MIP; ), SIMIP (Sea Ice MIP; ), and VIACSAB (Vulnerability, Impacts, Adaptation and Climate Services Advisory Board; ) – as these are not yet included in ES-DOC. There are of course many other “non-endorsed” MIPs such as ISA-MIP (the Interactive Stratospheric Aerosol MIP; ), which could also be documented with the ES-DOC system at some future time.

Figure  shows the sharing of experiments between MIPs. The importance of piControl, historical, AMIP, key scenario experiments (ssp245 and ssp585), and the idealized experiments (1pctCO2 and abrupt-4xCO2) is clear. These seven experiments form part of the protocol for many of the CMIP6 MIPs (Fig. ). The scope of the historical and piControl experiments is demonstrated by their connections to MIPs on the far edges of the plot in all directions.

There are other shared experiments too, which bring MIPs together around shared scientific goals: land-hist jointly defined and shared by LUMIP and L3SMIP; past1000 defined by PMIP forms part of VolMIP; piClim-control defined by RFMIP forms part of AerChemMIP; and dcppC-forecast-addPinatubo defined by DCPP forms part of VolMIP. By contrast, OMIP stands alone, sharing no experiments with other MIPs.

Experiments share forcing constraints, just as MIPs share experiments. Figure  shows the interdependence of the DAMIP experiments on common forcing constraints. Experiments are grouped near each other when they share forcing constraints. The dense network shown reflects the similarity of experiments within DAMIP and arises from a common design pattern or protocol in numerical experiment construction: a new experiment is a variation on a previous experiment with one (or a few) forcing changes. It is of course this “perturbation experiment” pattern which provides much of the strength of simulation in exposing causes and effects in the real world.

Unique modifications appear in Fig.  as forcing constraint nodes that are only connected to one or two experiments, which is also why the alternative forcing experiments hist-all-nat2 and hist-all-aer2 are placed further from the main body of the DAMIP network – they share fewer forcing constraints with the other experiments. However, they themselves are similar to each other as between them they share a number of unique forcing constraints.

The importance of the perturbation experiment pattern is further emphasized in DAMIP by noting that the three external experiments (piControl, historical and ssp245) account for 62 % of the DAMIP forcing constraints; five of the DAMIP experiments can be completely described by forcing constraints associated with these external experiments – being different assemblies of the same “forcing building blocks”. The key role of these building blocks is exposed by placing the DAMIP experiments into sets according to which of those external experiments is used for forcing constraints (Fig. ).

This framing of shared forcing constraints exposes some apparent anomalies. Why, for example, is hist-CO2 not in the historical set? The reasons for these apparent anomalies expose the framing of the experiments. In the historical experiment, greenhouse gas forcing is a single constraint, which includes CO2 and other well-mixed greenhouse gases. By contrast, hist-CO2 varies only CO2, with the other well-mixed greenhouse gases constrained to pre-industrial levels (and hence uses the piControl forcing constraints for those, with its own CO2 forcing constraint).

It would have been possible to avoid this sort of anomaly by constructing finer constraints in the case of historical, but this would have been at the cost of simplicity of understanding (and greater multiplicity in reporting as discussed below). There is a necessary balance between clear guidance on experiment requirements and reuse of such constraints to expose relationships between experiments.

One of the goals of the constraint formalism is to minimize the burden on modelling groups. Minimizing the burden of executing the CMIP6 experiments and the burden of documenting how the experiments were carried out (that is, populating the concrete part of the experiment definition, using the language of , as discussed in Sect. ). By clearly identifying commonalities between experiments, modelling groups can implement constraints once and reuse both the implementation and documentation across experiments.

Constraint “conformance” documentation is intended to provide clear targets for interpreting the differences between simulations carried out with different models. Given that differing constraints often define differing experiments, understanding why models give different results can be aided by understanding differences in constraint implementation (in those cases where there is implementation flexibility). Section  discussed some aspects of this from a constraint definition perspective.

One can then ask, how much reuse of constraints is possible? Figure , shows that a few forcing constraints are reused widely across CMIP6. These are the forcing constraints associated with the DECK and historical experiments and the prominent scenario experiments from ScenarioMIP which are used by numerous MIPs (Fig. ). It is with these forcing constraints that many deep connections between MIPs are made. From a practical perspective the wide application of these forcing constraints allows for considerable streamlining of the documentation burden on CMIP6 modelling groups. Beyond this core we see a smaller group of forcing constraints that are used by a few MIPs. For the most part these are forcing constraints associated with the less prominent scenario experiments from ScenarioMIP. The remainder of the forcing constraints are specific to just one MIP, and of these, 265 are only used once by a single experiment. Although this last group of forcing constraints is large in number, many groups will only make use of them if they happen to run the specific experiments to which they pertain.

History suggests there has been – and continues to be – divergent understanding of instructions for the expected duration of simulations (temporal constraints), often manifest by delivering “off by one” differences in the number of years of simulation. Such errors hamper statistical inter-comparison between simulations and can result in unnecessary effort (often expensive in human and computer time). The CMIP6 experiments have not been immune from this issue. Temporal constraints in the CMIP6 controlled vocabulary are defined in terms of a start year and a minimum length of simulation expressed in years. However, the publications by the CMIP6-endorsed MIPs often also include an end year which can be inconsistent with the minimum simulation length as described by the CMIP6-CV. The divergence in understanding generally occurs in the interpretation of the dates implied by a given start year and end year, specifically whether they refer to the beginning of January or the end of December.

A significant effort has been made by ES-DOC to identify these discrepancies and instigate their correction. ES-DOC temporal constraints unambiguously specify a start date, end date, and length for simulations and are a mandatory part of the ES-DOC experiment documentation. Despite these steps, there are still many cases where the MIPs of CMIP6 might have coordinated yet further and used the same temporal constraints for different experiments with essentially the same temporal requirements, such as those that begin in the present day and run to the end of the 21st century. These differences provide scope for further rationalization in future experiments and/or CMIP phases, leading to further simplification in analysis and savings in computer time.