Version 1.1 of the editorial of
The changes in the code and data policy are summarised as follows:
The requirement for authors to publish source code, unless this
is impossible for reasons beyond their control, is clarified. The
minimum requirements are strengthened such that all model code must
be made accessible during the review process to the editor and to
potentially anonymous reviewers. Source code that can be made public
must be made public, and embargoes are not permitted. Identical
requirements exist for input data and model evaluation data sets in
the model experiment descriptions. The scope of the code and data required to be published is
described. In accordance with Copernicus' own data policy, we now
specifically strongly encourage all code and data used in any
analyses be made available. This will have particular relevance
for some model evaluation papers where editors may now strongly
request this material be made available. The requirements of suitable archival locations are specified, along
with the recommendation that Zenodo is often a good choice.
In addition, since the last editorial, an “Author contributions”
section must now be included in all manuscripts.
The importance of open science and open data has become increasingly
recognised in the scientific community. Since the last editorial in
GMD
The short version of the argument is given by the motto of the Royal
Society: “nullius in verba” (“take nobody's word for
it”). Scientists who publish a result without publishing the calculations
undertaken to achieve that result are requiring the world to take them at
their word, which is inimical to the scientific method, and the only really
effective way to publish the calculations behind a computer model is to publish the
source code
Throughout this editorial, “must” means that the stated practice is required and that manuscripts which fail to comply will be rejected; “should” means that the practice is strongly encouraged, and authors will need to provide defensible reasons in cases where manuscripts do not comply.
Geoscientific models exist to enable computational experiments to be conducted. Those experiments take mathematical systems which are believed to
represent important features of some geoscientific system and
calculate the predicted behaviour of that mathematical system, in
order to gain insight into the real system, in order to either better
understand the system or to predict its response to forcing. The
hypothesis is as follows:
This mathematical system captures sufficient features of the system such that
useful predictions or insights about the geoscientific system can be drawn from it. The model code faithfully represents the mathematical and theoretical specification of
the model. The computation that was conducted used the code, input data and
configuration as intended. The analysis of the model output was appropriate and performed correctly.
Where this hypothesis has survived extensive experimental testing, one might describe the mathematical system as, in some
sense, a good model for the physical system. Establishing this places strong demands on the
modelling process. In particular there must be sufficient evidence to demonstrate the following:
Publication of source code most directly addresses the first of these, whilst the second and third can be greatly aided by properly specifying precise versions of code and input data in GMD manuscripts.
For example, suppose a scientist develops a new advection scheme and implements it in an atmospheric dynamics model. They publish a paper describing the new scheme and present the results of both idealised verification experiments and realistic simulations showing an improvement in forecast skill, but the code is not released. While this may appear to be a valuable contribution to modelling science, the complete algorithm including all approximations is needed in order to appreciate the sufficiency of the solution to the mathematical model. Even this would not be fully sufficient because all code contains unintentional errors (bugs), and even in the best code these will somewhat influence the results obtained. These issues are expanded on in the following paragraphs.
Space in journal papers is limited, as is the ability of readers to absorb information and the time authors can spend preparing publications. This means that it is normal for even simple models to be significantly underspecified in the paper itself. At the more complex end, it is not uncommon for an entire general circulation model of perhaps a couple of a million lines of code to be described in a 20–30-page paper (including verification and some evaluation). This obviously results in many details being omitted from the paper. Papers may omit discussion of boundary conditions or only deal with simplified cases. More seriously, model code frequently contains important details that are not mentioned in the paper, such as error traps that prevent the model from crashing. Recording every implementation detail in the paper would both be impractical and would undermine the paper as a mechanism for communicating the core ideas behind a model in a manner intelligible to readers.
Bugs occur in essentially all non-trivial code. Good software
engineering practices such as code review
Both underspecification and bugs result in model code whose behaviour can differ from the mathematical system that the author and/or reader believes is being run. Suppose a reader finds a surprising result in a published simulation using this model. What should they do? Assuming enough resources the reader could attempt to re-write the model: except that underspecification prevents this. Even if they achieve this or employ one or more different models of the same physical system, the most they will be able to achieve is to conclude that another model fails to reproduce the original result. Even given two codes attempting to implement the same algorithm, there is no guarantee that the results will be the same, and without source code it is impossible to establish the underlying cause of the differences. With the source code, however, readers can gain much deeper knowledge of the model than can ever be described in the paper.
The hypothesis highlighted above is easily adapted to all the other forms of coded products described in GMD papers, such as data assimilation systems, frameworks, databases and model evaluation tools. Although not always emulating a physical system, in these products there is an underlying mathematical structure that is realised in code, and it must be demonstrated to work as intended in just the same way as a model. For model experiment descriptions, the situation is very simple. The primary purpose of these papers is to enable modelling communities to perform the same experiments. Therefore, everything required to run the experiment must be provided, apart from the model itself.
In order for the reader to have some confidence that the hypotheses above hold, it is not sufficient that the source code is provided. It is also necessary to have access to all of the input data and to know all of the steps which were taken from raw data to points on graphs or numbers in tables. This also implies that all model configuration files are provided.
A particular challenge to understanding and reproducing results occurs where model inputs or outputs have been manually processed by an author. This frequently occurs when figures are produced interactively. Unfortunately, this breaks the provenance chain between the model and the paper: nobody, not even the author, can definitively know what processing was done to the data and therefore how the results came about. The only remedy for this is for authors to consistently ensure that there is no manual processing of the data: models are run by a script, and all pre- and post-processing is scripted. These scripts themselves should then be archived and cited from the paper. All figures and tables must be scientifically reproducible from the scripts.
The preceding sections make the case that publication of source code and associated data is a necessary part of the scientific method. Nonetheless, it is the case that the authors of some papers may not be able to publish their code or data. For example, this can occur when their institution owns the copyright and refuses to allow the code or data to be licensed in a manner which enables open archiving, or it may be the case that the paper authors are dependent on code or data whose copyright is owned elsewhere and for which they do not have a licence to redistribute. In particular, some of the current Earth system models have restricted licences controlled by large institutions. In other cases, model input or output may simply be too large to be uploaded to any open archival system that is available to the authors.
This presents a challenge: should GMD insist on publication of source
code and data and therefore not publish papers about some of the most important models in
the geosciences, or should it accept such papers even though the rigour of
those papers is compromised by the lack of code? At the time of writing, the
GMD editors considered that the balance falls on the side of allowing
publication. However this does compromise the scientific standards of the
journal, so the circumstances in which source code publication will not be
required must be drawn as narrowly as is feasible. All manuscripts
must at a minimum provide confidential access to the code and
data developed in the manuscript for the editor and reviewers in order to enable peer review
(see Appendix
This position is broadly consistent with the American Geophysical Union
publications data
policy
In the case where the new code and data described in the paper are not restricted but are part of a larger code and data structure that has other restricted elements, it is still possible to satisfy the GMD requirements by making the new parts of the code and data available. As the result is usually not a coherent model which can be expected to compile, authors sometimes prefer to upload these code fragments to the supplement rather than to a repository. Authors may have to remove restricted elements from their model code base, but in the meantime this remains an acceptable if somewhat unsatisfactory solution.
Recently, some authors have offered to provide public code access only on acceptance of the final manuscript or after a defined period, such as 12 months. In these circumstances, it is clear that the authors are able to provide code access. By preventing public access to the code during open review and/or during the immediate period after publication, they are impeding the scrutiny of their work at the most critical points in the publication process. Having determined that source code publication is a necessary part of the publication process, GMD does not permit embargoes on code release. It is the opinion of the GMD editors that if the code is not ready, then neither is the manuscript. Therefore, if the code is not subject to licensing or other restrictions preventing its ultimate release, it must be made available to the editor and reviewers upon submission.
A related issue to embargoing is raised by manuscripts which point to code located on a website, accompanied by a promise in the cover letter that the code will be properly archived when the final paper is accepted. This approach compromises the review process for two reasons. First, at the core of the open review process of European Geosciences Union journals is the idea that the submitted manuscript and the review process are openly and persistently available to scrutiny. The version of the code which matches the submitted manuscript directly influences that process, so it must remain available to anyone who wishes to examine the review of the manuscript.
Second, the whole review process, including initial editor review, external referees and executive editor intervention when needed, is in essence a protracted quality control process for the scientific content of the manuscript but also for the technical requirements such as code and data archiving. If authors are permitted to delay compliance with technical requirements, then this effectively disables most of the layers of this quality control process. For this reason, the full archival requirements must be met by manuscripts on submission, and handling topical editors should require revised submissions by authors until these requirements are met. This provides reviewers and executive editors with the opportunity to act as a backstop check during the review process, thereby minimising the chance of a final paper being published which fails to meet the code and archiving requirements.
The key objection that some authors hold to archiving on submission is that it can result in old, uncorrected, versions of the code being persistently available online. There are mechanisms available to assuage this concern. First, the code and data availability section in the manuscript can and should identify the preferred download location for the latest model version, in addition to citing the archived version corresponding to the given publication. Second, archives such as Zenodo support the archival of successive versions of data sets and flag to the reader that a newer version is available. Further, the archive metadata can also direct readers to the preferred download location.
A possible response to the issues caused by lack of access to the source code is to go back to the authors and ask them to provide code or otherwise assist in investigating a surprising result. The success of this approach is dependent on the authors being willing and able to assist. The latter is a particularly difficult problem: if code has not been curated sufficiently well, it may not be possible to recover the exact version used in a paper; the scientist who did the work may have moved on, or the resources (human or computational) required for assistance may not be available.
This undermines the role of a journal as a persistent, public and definitive archive of scientific results. In order for journals to fulfil this function, it must be possible for readers to trace the provenance of the results presented without further assistance by the authors. For source code (and other data) to fulfil this role, the data archive must be as persistent, public and definitive as the journal itself. Relatively small models, data sets or documents can often be uploaded as supplements to the model and stored alongside the paper itself. However this approach is impractical for larger data sets. It is also not always clear that the very high standards of long-term preservation that journal publishers provide for articles are also applied to supplements.
There are three highly desirable features of an archival system which is to
be used for the code and other data on which a journal publication depends:
Zenodo is a scientific data archive funded by the European Union and
hosted by CERN. It has an institutional guarantee of at least 20 years
of future support and has policies restricting withdrawal of deposited
data
On occasion, GMD requires authors to archive grey literature, for example, technical
reports about a previous model version. Where these are simply a single document, Zenodo may
not be the most convenient archival system. In these cases the arXiv
preprint server is a good choice
At the time of writing, Zenodo had an upload limit of 50 GB. This is more
than enough for source code and indeed for the input data for many GMD
papers. However some papers, especially evaluation papers for complex
models, may require much more than this. Because the archival of large
data sets is expensive, there is no generic free solution to this case:
the appropriate repository may depend on the application area of the
model in question, the source of funding for the research or the
location of the authors' institutions. The requirements of
Sect.
Hosting source code and other data on the authors' institutions' websites fails to satisfy the requirements for irrevocability and persistence of reference. Even large institutions periodically refresh their web presence with the result that URLs change, and data from long-finished projects may not be preserved. Of course data archives also have web interfaces, and some authors work for institutions that host suitable data archives. Authors are not debarred from using a data archive which complies with GMD policy simply because they are affiliated with the archive's host organisation. However by the same token, the fact that an organisation hosts a suitable archive does not mean that organisation's entire web presence can be considered an archival location.
In the last few years, many authors have submitted manuscripts whose
code availability sections consist of direct links to online revision
control systems, either institutionally hosted or online services such
as GitHub
The main purpose of this editorial is to provide greater clarity on the code and data policy at GMD. The most significant change is that all source code (including input data for model description papers and data sets for evaluation data set description papers) must be made available to both the editor and reviewers.
In Appendix A to this editorial we include the revised code and data policy information, which replaces the previous GMD-specific text on the code and data policy web page. There are also some small changes made to the paper types page, the revised version of which is included in Appendix B.
Every paper must include a section at the end of the paper before the
“Acknowledgements” entitled “Code availability” or “Code and data
availability” as appropriate.
This section must include citations for the persistent public
archives of the precise versions of all of the code and data
associated with the paper. The generic means to access other versions
of the code and data as well as the licence of the code should also
be explained. The licence should conform to
the Open Source
Definition Where the authors cannot, for reasons beyond their control,
publicly archive part or all of the code and data associated with a
paper, they must clearly state the restrictions. They must also
provide confidential access to the code and data for the editor and
reviewers in order to enable peer review. The arrangements for this
access must not compromise the anonymity of the reviewers. All
manuscripts which do not make code and data available at this level
are to be rejected. Where only part of the code or data is subject to
these restrictions, the remaining code and/or data must still be
publicly archived. In particular, authors must make every endeavour to
publish any code whose development is described in the manuscript.
Code and data access must be provided at the time that the discussion
paper is submitted. Embargoes, whether pending acceptance or for a defined
period, are not acceptable.
The code and data associated with a paper which are subject to the
above requirements include, depending on the paper type, the following:
the source code for the complete model or module or other coded
product described in the paper (must be provided for model
description, development and technical, and methods for assessment
paper types); the manual and any other model documentation (applies to model
description, development and technical, and methods for assessment, to the
extent the editor considers applicable); all configuration files, boundary conditions, and input data (must be
provided for experiment description papers and any other papers in
which results from model runs are reported); data sets for forcing of models or comparison with model output
(must be provided for papers describing such data sets or for papers
in which model output are compared with such data); preprocessing, run control and postprocessing scripts covering every
data processing action for all the results reported in the paper
(applies for all papers, to the extent the editor considers applicable).
In every case, the citation from the paper must identify the exact version
of the code and/or data used.
Although the code and data will not be reviewed formally, the editor and reviewers are free to make general comments on any code or data, if they so wish. During the review process, the ease of model download, compilation, and running of test cases may be assessed.
A frozen version of
the code and data as developed in the paper must be archived. Usually, a
third-party archive is preferable. In some cases, such as when the
code is a fragment from a larger model, authors may include the code in the supplement to the paper. Third-party archives must have the following:
institutional support providing reasonable confidence that the
material will remain available for many years/decades mechanisms preventing the depositor of the material from unilaterally
removing it from the archive mechanisms for identifying the precise version of the material referred
to in a persistent way. This will usually be a DOI.
Where code and data change during the revision process of the manuscript,
the updated versions must also be archived. Authors must take care that the
results in revised manuscripts are correctly associated with the
corresponding archived data (with different DOIs referenced in the
submitted and final manuscripts in cases where data have changed).
Many GMD authors find Zenodo
Project or institution websites and online revision control sites such
as GitHub
The following code and data availability section meets the requirements of
this policy for papers focussed on development of models or
development of methods for assessment of models. Other wordings are, of course, possible so long as the required
information is all present. For larger models it is very helpful if
authors can identify the location of the main parts of the code that are discussed in the
manuscript. For experiment description papers, evaluation papers, and
some technical and development papers where
details for a variety of different data sets or models are required, the section
will be considerably longer. The current version of
In line with the FORCE11 Joint Declaration of Data Citation Principles, the
data citations should appear in the bibliography and be referenced in the
text in the same way as other publications
Below is the manuscript types web page content. Changes from Editorial 1.1 are in bold font.
In the following, “must” means that the stated practice is required and that manuscripts which fail to comply will be rejected; “should” means that the practice is strongly encouraged, and authors will need to provide defensible reasons in cases where manuscripts do not comply.
Code and/or data availability sections must be included in all papers
and should be located at the end of the article, after the conclusions,
and before any appendices or acknowledgements.
There are seven different manuscript types accepted at GMD. During the
submission process, authors will need to select the type which most
closely matches the aims of their manuscript. The types are as follows:
model description papers development and technical papers methods for assessment of models model experiment description papers model evaluation papers review and perspective papers corrigenda.
Updates: Minor version updates or correction of actual errors in a model, model development or experiment protocol should be submitted as a regular submission within one of the standard manuscript types. Authors may request that these form part of a model special issue including the previously published papers.
Model description papers are comprehensive descriptions of numerical
models which fall within the scope of GMD. The papers should be
detailed, complete, rigorous, and accessible to a wide community of
geoscientists. In addition to complete models, this type of paper may
also describe model components and modules, as well as frameworks and
utility tools used to build practical modelling systems, such as
coupling frameworks or other software toolboxes with a geoscientific
application. The GMD definition of a numerical model is generous,
including statistical models, models derived from data (whether model
output or observational data), spreadsheet-based models, box models,
1-dimensional models, through to multi-dimension mechanistic models.
The main paper must give the model name and version number (or
other unique identifier) in the title. The publication should consist of three parts: the main paper, a
user manual, and the source code, ideally supported by some summary
outputs from test case simulations. The main paper should describe both the underlying scientific
basis and purpose of the model and overview the numerical solutions
employed. The scientific goal is reproducibility: ideally, the
description should be sufficiently detailed to in principle allow for
the re-implementation of the model by others, so all technical details
which could substantially affect the numerical output should be
described. Any non-peer-reviewed literature on which the publication
rests should be either The model web page URL, the hardware and software requirements and
the licence information should be given in the text. If papers are
describing subsequent development to a paper already published in GMD,
The model description should be contextualised appropriately. For
example, the inclusion of discussion of the scope of applicability and
limitations of the approach adopted is expected. Examples of model output should be provided, with evaluation
against standard benchmarks, observations, and/or other model output
included as appropriate. In this respect, authors are expected to
distinguish between verification (checking that the chosen equations
are solved correctly) and evaluation (assessing whether the model is a
good representation of the real system). When Although the source code and user manual will not be reviewed
formally, the editors and reviewers are free to make general comments
on the code if they so wish. During the review process, the ease of
model download, compilation and running of test cases may be assessed.
These papers describe technical developments relating to model improvements such as the speed or accuracy of numerical integration schemes as well as new parameterisations for processes represented in modules. Also included are papers relating to technical aspects of running models and the reproducibility of results, e.g. assessments of their performance with different compilers, or under different computer architectures. In addition, papers focussing on data assimilation are welcome. Development and technical papers usually include a significant amount of evaluation against standard benchmarks, observations, and/or other model output as appropriate.
In the case where new code is described in the paper, this is subject to the same availability requirements as for complete model descriptions. The code should be made available, and a model availability paragraph must be included.
If the model development relates to a single model, then the model name and the version number must be included in the title of the paper. If the main intention of an article is to make a general (i.e. model independent) statement about the usefulness of a new development, but the usefulness is shown with the help of one specific model, the model name and version number must be stated in the title. The title could have a form such as, “Title outlining amazing generic advance: a case study with Model XXX (version Y)”.
Methods for assessment of models include work on developing new metrics for assessing model performance and novel ways of comparing model results with observational data. Also included are discussions of novel methods for data analysis, visualisation with relevance to geoscientific modelling, or the application of existing techniques to this field. These papers may also be theoretical, in which case an example implementation should be provided as supplementary information. They may also be based on the description of a fully fledged software tool.
The process of analysing model output for comparison with data may involve algorithms similar to those implemented in complex numerical models. In these cases, model output is input to another model in order to produce output comparable to observed quantities. Papers describing these algorithms may be submitted as either methods for model assessment or model description papers.
Descriptions of software tools are subject to the same criteria as model descriptions (name and version must be identified in the title, code must be supplied for the peer-review process, etc.), and a code availability paragraph must be included in the manuscript.
Model experiment description papers contain descriptions of standard
experiments for a particular type of model, such as might be used in a
MIP
Papers describing data sets designed for the support and evaluation of model simulations are within scope and included in this paper type. These data sets may be syntheses of data which have been published elsewhere. The data sets must also be made available, and any code used to create the syntheses should also be made available.
Model evaluation is an important component of most GMD papers. Model
development papers in particular often include a large proportion of
evaluation. Typically, this comprises a comparison of the performance of
different model configurations or parameterisations. In some cases, the
evaluation is sufficiently substantial that a stand-alone paper is
required. In this case it is required that the model, model development,
or model experiment has already been described in another paper (or that
the description is also under review).
It is, however, common for pure evaluation papers to contain substantial conclusions about geoscience rather than about models, and such papers are not suitable for submission to GMD. These are more likely to reach the appropriate audience in those EGU journals which publish scientific results related to the GMD subject areas.
Review and perspective papers summarise the status of knowledge and outline future directions of research within the scope of the journal.
Before preparing and submitting a review article, please contact the executive editors. A code and/or data availability section must be included. By default, the code and data availability requirements for models, experiments, code and data discussed in review papers are the same as for the other paper types, but in some cases deviations from this standard may be appropriate (for example, authors may need to discuss some code or data from external sources, for which they have no means of gaining or granting access). This should be discussed with the executive editors prior to submission of the paper.
Corrigenda correct errors in preceding papers. The manuscript title is as follows: Corrigendum to “TITLE” published in JOURNAL, VOLUME, PAGES, YEAR. Please note that corrigenda are only possible for final revised journal papers and not for the corresponding discussion paper. Corrigenda should only be used for correcting errors in the papers and not for those occurring in the model development being described.
The guidelines presented in this paper have been approved by all GMD executive editors. Most of text was written by DAH and JCH, with all other editors (AK, DMR, and RS) contributing to the discussion of the issues presented here at some length via frequent email communication. This editorial presents what we believe to be an achievable best practice for the journal over the next few years and as such does not represent the ideal for any of the executive editors.
The GMD executive editors gratefully acknowledge the GMD topical editors for their hard work, support and helpful comments on the manuscript. David A. Ham was supported by a United Kingdom Natural Environment Research Council Independent Research Fellowship [grant no. NE/K008951/1]. Didier Roche was supported by the Centre national de la recherche scientifique (CNRS) and by the Vrije Universiteit Amsterdam.