Editorial : The publication of geoscientific model developments v 1

In 2008, the first volume of the European Geo- sciences Union (EGU) journal Geoscientific Model Develop- ment (GMD) was published. GMD was founded because we perceived there to be a need for a space to publish compre- hensive descriptions of numerical models in the geosciences. The journal is now well established, with the submission rate increasing over time. However, there are several aspects of model publication that we believe could be further improved. In this editorial we assess the lessons learned over the first few years of the journal's life, and describe some changes to GMD's editorial policy, which will ensure that the mod- els and model developments are published in such a way that they are of maximum value to the community. These changes to editorial policy mostly focus on improv- ing the rigour of the review process through a stricter re- quirement for access to the materials necessary to test the behaviour of the models. Throughout this editorial, "must" means that the stated ac- tions are required, and the paper cannot be published without them; "strongly encouraged" means that we encourage the action, but papers can still be published if the criteria are not met; "may" means that the action may be carried out by the authors or referees, if they so wish. We have reviewed and rationalised the manuscript types into five new categories. For all papers which are primarily based on a specific numerical model, the changes are as fol- lows: - The paper must be accompanied by the code, or means of accessing the code, for the purpose of peer-review. If the code is normally distributed in a way which could compromise the anonymity of the referees, then the code must be made available to the editor. The ref- eree/editor is not required to review the code in any way, but they may do so if they so wish.


Introduction
The journal Geoscientific Model Development (GMD) was started in 2008 as a response to the need to make possible full and formal publication of model descriptions in the geosciences.The journal allowed, for the first time, for models to be published such that they are scientifically reproducible, for model developments to be traceable through the literature, for model descriptions to be accessible to all, and for model descriptions to be formally peer reviewed (GMD Executive Editors, 2013, hereafter Editorial 1.0).Since the journal was started, the movement towards a greater openness in science has continued, and ideas introduced by GMD have also been adopted elsewhere.In Editorial 1.0 it was observed that, since the start of GMD, it had become less difficult to publish model descriptions in the more general scientific literature.The response to the publication of Editorial 1.0 was that other journals started to increase the requirement to make code or data more generally available (Nature, 2014).
The journal continues to grow at a healthy rate.has been found to be useful by the community.Likewise, a paper that describes a community modelling experiment protocol should also be highly cited, if the experiment itself is successful.However, GMD also encourages the submission of other types of papers that are less likely to be widely cited, such as small updates to models, reporting of errors in models and technically detailed papers focussing on the numerics of a particular model or parameterisation.It is interesting to observe how this balance plays out.Table 1 shows the evolution of the impact factor since the start of the journal.(The impact factor is the average number of citations attained in a given year of papers published in the previous 2 years.)It is possible that variation in the impact factor may be linked to the timing of the IPCC (Intergovernmental Panel on Climate Change) deadlines, which dictate model development phases in the climate sciences.Around 2011-2012, a number of papers focussing on major GCMs (general circulation models) were published and these are very highly cited.In contrast, in 2014 the ratio of papers with no citations increased considerably, as more smaller, more technical papers were submitted.While papers describing incremental model developments are encouraged at GMD, papers must include a tangible and potentially useful advance related to model development.With the growing submissions to the journal, editors are now encouraged to carefully consider whether a manuscript offers a genuine advance before sending it through to peer review.Two years after the last editorial, it is now time to include some minor updates and clarifications to the peerreview process.These are highlighted in the abstract and discussed in the sections below.The new Framework for GMD manuscript types (Appendix A), the GMD code and data policy (Appendix B), and the Publication guide (Appendix C) are included.
As in Editorial 1.0, throughout this editorial, "must" means that the stated actions are required, and the paper cannot be published without them; "strongly encouraged" or "should" means that we encourage the action, but papers can still be published if the criteria are not met; "may" means that the action may be carried out by the authors or reviewers, if they so wish.

Changes to the peer-review framework for model publication
Every scientific paper is different, and inclusion of new techniques and practices can often result in papers that are difficult to classify into any one of the GMD manuscript types.Nevertheless, there are some areas in which some clarification is now required in order to make the official peer-review criteria consistent both with the aims of the journal for the future, and with the practical realities of the kinds of papers that are presently accepted and rejected at the journal.In this section we explain the rationale behind the changes.The new versions of the Framework for GMD manuscript types (hereafter the Framework) and Publication guide are published in the appendices.

Code availability
In Editorial 1.0 the requirement to make code available was introduced.In cases where legal restrictions mean that the code cannot be made publicly available, the code must still be made available to at least the editor.It must also be accessible to the reviewers if legally possible.This is clearly stated in the abstract of Editorial 1.0, but as the wording is slightly unclear in Framework 1.0, it has now been updated for version 1.1.In addition, in cases where the code is not made publicly available, the authors must grant access to the topical editor, who may reject the paper if this access is not given at some point during the peer-review process.The Publication guide has been updated to reflect this change.What was perhaps not made so clear in Editorial 1.0 was the executive editors intention that all papers which describe model developments should make the code available, no matter what kind of "type" the manuscript falls into.In addition, papers which describe any kind of new code that may be useful to the scientific community should by default make the code available.We find that, despite Framework 1.0 already requesting authors to make code available for all manuscript types, the authors have not always considered this issue before submitting their papers.For this reason, the "Code availability" paragraph is to become mandatory for all manuscript types.For model experiment description papers or other papers where the required information is not the actual model code, the title may be changed to something more appropriate such as, "Data availability".The code and/or data availability sections should be located at the end of the article, after the conclusions but before the appendices and acknowledgements.

The role of model evaluation
Model evaluation is an important aspect of many GMD papers.The evaluation may be against data or against other models or derived metrics.All new models or model versions need some kind of evaluation before they can be considered fit for purpose.Thus, most model description papers will include some evaluation as part of the proof that the model behaves as conceived.Development and technical papers often include evaluation to show the value of the proposed changes to the models.Model assessment methods papers should normally include evaluation as a demonstration of the value of the assessment method being used.Model experiment description papers may be submitted either before or after the experiment is performed.In the latter case, overview results demonstrating that the experiment setups work to produce the expected results should be included.
Given that evaluation is such an important component of almost all GMD papers, we have decided to remove the explicit mention of the model evaluation manuscript type within the "technical development and evaluation" paper type.The model evaluation manuscript type is moved to a new section and the circumstances under which a paper that concerns only model evaluation may be submitted to GMD are specified, and the general role of evaluation is clarified throughout Framework 1.1.

Kinds of models accepted at GMD
Exactly what constitutes a geoscientific model is somewhat subjective.A large variety of models are submitted to GMD, including statistical models, models derived from data (whether model output or observational data), spreadsheetbased models, box models, 1-dimensional models, through to multi-dimension mechanistic models such as Earth system models.All these types of models are welcome at GMD and this is clarified in Framework 1.1.
Papers should, however, present a significant advance, and be clearly focussed.They should be written in such a way as to be accessible and potentially useful to other members of the geoscientific modelling community.Trivial, poorly written or otherwise incomprehensible papers will be rejected at the initial review stage.Development papers or papers describing model errors or version updates may contain rather small or simple extensions to existing models, but if these papers are done well, then the discussion, comparison of model versions and evaluation against data results in such papers being substantial contributions.The Publication guide is updated to reflect this issue.

Papers focussing on data assimilation
Data assimilation schemes for models can be highly complex with a substantial amount of code.For example, in the case of an adjoint-based method, the data assimilation scheme uses a backwards running version of the model.Thus, it is just as important that data assimilation schemes are properly documented as the models themselves.Data assimilation papers are already regularly published in GMD, and now we explicitly include them in the development and technical manuscript types.

Papers focussing on data
It has become necessary to clarify the ways in which observational data may be published in GMD.This topic was not previously mentioned in the peer-review criteria.For such papers, authors are encouraged to also consider open-access data journals (Candela et al., 2015) which are more suitable than GMD for the publication of new observational data.There is, however, an area of development directly relevant to numerical modelling which may suit publication in GMD.This is the development of data sets designed specifically either for use as boundary conditions for model experiments or for direct comparison with model output as a method of model evaluation.Typically, these data sets will be syntheses of previously published data, processed so as to be directly compatible with model inputs and outputs in terms of model variables and gridding.A particular example could be the presentation of boundary conditions for one or more of the model intercomparison projects (MIPs).Since such papers are likely to be of high interest to the geoscience modelling community, they may be submitted to GMD.As they are often accompaniments to specific model experiments, these papers are now included within the model experiment description manuscript type and the Framework is updated to reflect this.

Plagiarism
As part of the Copernicus Publications' review system, for each new submission, a "similarity report" is automatically produced and made available to the editor.As described more fully in the Publication guide, there are two main issues for GMD related to plagiarism: the generally acceptable practice of copying text from internal documents or reports, and the more problematic copying from previously peer-reviewed publications of text blocks of model descriptions.

Conclusions
We have implemented some minor changes to the peerreview policy at GMD. Mostly these may be considered clarifications to Editorial 1.0.The requirements for making code accessible have been updated and are further clarified by the provision of a separate appendix defining GMD's code and data policy.We also define the limited conditions under which data sets derived from observations may be published, under what conditions a separate evaluation paper may be appropriate, and explicitly include descriptions of data assimilation schemes into the development paper type.The Publication guide is updated to reflect the changes in the Framework and includes additional guidance on the assessment of models at the initial review stage.

Appendix A: Framework for GMD manuscript types v1.1
Much of this text is the same as that in Appendix A of Editorial 1.0.Additional or changed sentences are marked in bold.
This framework is maintained online at the GMD website (http://www.geoscientific-model-development.net/), on the Manuscript types page.
Here, "must" means that the stated actions are required, and the paper cannot be published without them; "strongly encouraged" or "should" means that we encourage the action, but papers can still be published if the criteria are not met; "may" means that the action may be carried out by the authors or reviewers, if they so wish.

A1 Code and data availability
Inclusion of Code and/or data availability sections is mandatory for all papers and should be located at the end of the article, after the conclusions, and before any appendices or acknowledgments.For more details refer to the code and data policy (Appendix B, and also available on the GMD website).

A2 Model description papers
Model description papers are comprehensive descriptions of numerical models which fall within the scope of GMD.The papers should be detailed, complete, rigorous, and accessible to a wide community of geoscientists.In addition to complete models, this type of paper may also describe model components and modules, as well as frameworks and utility tools used to build practical modelling systems, such as coupling frameworks or other software toolboxes with a geoscientific application.The GMD definition of a numerical model is generous, including statistical models, models derived from data (whether model output or observational data), spreadsheet-based models, box models, 1dimensional models, through to multi-dimension mechanistic models.
-The main paper must give the model name and version number (or other unique identifier) in the title.
-The publication should consist of three parts: the main paper, a user manual, and the source code, ideally supported by some summary outputs from test case simulations.
-The main paper should describe both the underlying scientific basis and purpose of the model and overview the numerical solutions employed.The scientific goal is reproducibility: ideally, the description should be sufficiently detailed to in principle allow for the reimplementation of the model by others, so all technical details which could substantially affect the numerical output should be described.Any non-peer-reviewed literature on which the publication rests should be uploaded as supplementary information.
-The model webpage URL, the hardware and software requirements and the license information should be given in the text.If papers are describing subsequent development to a paper already published in GMD, they will be electronically linked to the previous version(s) in a special issue, and an overview webpage will be created.
-The model description should be contextualised appropriately.For example, the inclusion of discussion of the scope of applicability and limitations of the approach adopted is expected.
-Examples of model output should be provided, with evaluation against standard benchmarks, observations, and/or other model output included as appropriate.In this respect, authors are expected to distinguish between verification (checking that the chosen equations are solved correctly) and evaluation (assessing whether the model is a good representation of the real system).Where evaluation is very extensive, a separate paper focussed solely on this aspect may be submitted, as described in Sect.A6.
-All papers must include a section, at the end of the paper, entitled "Code availability".Here, either instructions for obtaining the code, or the reasons why the code is not available should be clearly stated.It is preferred for the code to be uploaded as a supplement or to be made available at a data repository with an associated DOI (digital object identifier) for the exact model version described in the paper.Alternatively, for established models, there may be an existing means of accessing the code through a particular system.In this case, there must exist a means of permanently accessing the precise model version described in the paper.In some cases, authors may prefer to put models on their own website, or to act as a point of contact for obtaining the code.Given the impermanence of websites and email addresses, this is not encouraged, and authors should consider improving the availability with a more permanent arrangement.After the paper is accepted the model archive should be updated to include a link to the GMD paper.
-When copyright or licensing restrictions prevent the public release of model code, or in the cases where there is some other good reason for not allowing public access to the code, topical editors must still be given access to the model code.Access must also be granted to the reviewers whilst preserving their anonymity, if this is legally possible.-Although the source code and user manual will not be reviewed formally, the editors and reviewers are free to make general comments on the code if they so wish.During the review process, the ease of model download, compilation and running of test cases may be assessed.

A3 Development and technical papers
These papers describe technical developments relating to model improvements such as the speed or accuracy of numerical integration schemes as well as new parameterisations for processes represented in modules.Also included are papers relating to technical aspects of running models and the reproducibility of results, e.g.assessments of their performance with different compilers, or under different computer architectures.In addition, papers focussing on data assimilation are welcome.Development and technical papers usually include a significant amount of evaluation against standard benchmarks, observations, and/or other model output as appropriate.
In the case where new code is described in the paper, this is subject to the same availability requirements as for complete model descriptions.The code should be made available, and a model availability paragraph must be included. If

A4 Methods for assessment of models
Methods for assessment of models include work on developing new metrics for assessing model performance and novel ways of comparing model results with observational data.Also included are discussions of novel methods for data analysis or visualisation with relevance to geoscientific modelling, or the application of existing techniques to this field.These papers may also be theoretical, in which case an example implementation should be provided as supplementary information.They may also be based on the description of a fully fledged software tool.
The process of analysing model output for comparison with data may involve algorithms similar to those implemented in complex numerical models.In these cases, model output is input to another model in order to produce output comparable to observed quantities.Papers describing these algorithms may be submitted as either methods for model assessment or model description papers.
Descriptions of software tools are subject to the same criteria as model descriptions (name and version must be identified in the title, code must be supplied for the peerreview process, etc.), and a code availability paragraph must be included in the manuscript.

A5 Model experiment description papers
Model experiment description papers contain descriptions of standard experiments for a particular type of model, such as might be used in a MIP.Configurations and overview results of individual models can also be included as well as descriptions of the methodology of experimental procedures such as ensemble generation.Such papers should include the discussion of why particular choices were made in the experiment design and sample model output.In the case of papers describing MIPs, they should explain any specific project protocols, should highlight differences in the application of the protocol by the different groups, and should include sufficient descriptions/figures of model results to give an overview of the project.For model experiment description papers, similar version control criteria apply as to model description papers: the experiment protocol should be given a version number; boundary conditions should be given a version number and uploaded or made otherwise available; a data availability paragraph must be included in the manuscript; and links to the GMD paper should be included on the experiment website.Papers describing data sets designed for the support and evaluation of model simulations are within scope.These data sets may be syntheses of data which have been published elsewhere.The data sets must also be made available, and any code used to create the syntheses should also be made available.

A6 Model evaluation papers
Model evaluation is an important component of most GMD papers.Model development papers in particular often include a large proportion of evaluation.Typically, this comprises a comparison of the performance of different model configurations or parameterisations.In some cases, the evaluation is sufficiently substantial that a stand-alone paper is required.In this case it is required that the model, model development, or model experiment has already been described in another paper (or that the description is also under review).The authors should provide the citation of the description paper in the evaluation manuscript itself and also in the letter to the editor when submitting an evaluation manuscript.If the description is in GMD then there is the possibility of linking the papers, either in the form of a companion paper (e.g.Part 1 and Part 2), or as part of a special issue devoted to a particular model or experiment.
It is, however, common for pure evaluation papers to contain substantial conclusions about geoscience rather than about models, and such papers are not suitable for submission to GMD.These are more likely to reach the appropriate audience in those EGU journals1 which publish scientific results related to the GMD subject areas2 .

A7 Updates
Minor version updates or correction of actual errors in a model, model development or experiment protocol should be submitted as a regular submission within one of the standard manuscript types, and authors may request that these form part of a model special issue including the previously published papers.

A8 Corrigenda
Corrigenda correct errors in preceding papers.The manuscript title is as follows: Corrigendum to "TITLE" published in JOURNAL, VOLUME, PAGES, YEAR.Please note that Corrigenda are only possible for final revised journal papers and not for the corresponding discussion paper.Corrigenda should only be used for correcting errors in the papers and not for those occurring in the model development being described.
-Although not recommended, and authors will typically be requested to improve on this, it is also possible to provide the code and/or data only upon request via a given point of contact.
-If the authors cannot or do not wish to make the code and/or data public (e.g.copyright or licensing restrictions), the reasons must be clearly stated.Note that, for the purpose of the review, the code and/or data must still be made available to the editor.Access must also be granted to the reviewers whilst preserving their anonymity, if this is legally possible.
Although the source code and user manual will not be reviewed formally, the reviewers are free to make general comments on any code or data, if they so wish.During the review process, the ease of model download, compilation and running of test cases may be assessed.

Appendix C: Publication guide v1.1
In this Publication guide, we concentrate on the points of the peer-review process most relevant to GMD editors, reviewers and authors.General guidance on the interactive public peer-review system is available elsewhere on the journal's website.

C1 Initial submission
All authors, reviewers, and editors should make themselves aware of the journal manuscript types (http://www.geoscientific-model-development.net/about/manuscript_types.html).On initial submission, the editor first reads the manuscript to check that it is within scope and that it is of an appropriate quality to enter the peer-review process.Manuscripts must contain a tangible advance in the field of geoscientific modelling.Those which contain poorly supported scientific results should be rejected at this stage.Next, a few further checks need to be made: -Editors must check that the title of the paper includes the name and version number of the model or experiment discussed in the paper, where appropriate.
-All papers must contain a code availability (or equivalent for model experiment description papers) paragraph, which states how the code may be obtained.For those cases where authors cannot or do not wish to make the code available once the manuscript is published, the reasons for this must be clearly stated in the manuscript.
-In the case of a model or development description where code is not made publicly available for licensing or copyright restrictions reasons, the authors must grant access to the topical editor, who may reject the paper if this access is not given at some point during the peer-review process.In other cases the editors and reviewers are strongly encouraged to access the code through the means specified in the availability paragraph.The editor may need to liaise with the authors to make the model code available to the reviewers without compromising the reviewers' anonymity.
In the present system, finding the best solution to the points above is often reached via an amicable agreement in accord with GMD's aims through personal email exchange between editor and authors.Then, once a solution is agreed upon, the editor is able to enter their initial decision into the Copernicus system (e.g.technical corrections or minor revision).
At the initial stage the editor must also study the similarity report, if present.There are two points particular to GMD to consider here.
-Some models and model developments are published as institutional internal documents or reports.For reasons of the traceability of model descriptions and code, GMD actively encourages authors to improve these "grey literature" documents so that they comply with GMD peer-review criteria and then to submit them for peer review at GMD.In the case where the internal document is available online, the similarity report metric may be very high.In such cases the editor does of course have to investigate the status of the earlier report, so, to avoid problems later on, it is helpful if the authors include information about any such other documents in their letter to the editor.
-It is not uncommon to find in a model description submitted to GMD small paragraphs copied from another publication with only light editing.Often the other paper has a few authors in common with the authors of the submitted manuscript.While such self-plagiarism is not generally perceived as being as serious an offence as stealing someone else's results, it still needs to be remedied.If authors do not wish to rephrase the work in their own words, one obvious solution is for them to explicitly quote the other work with a citation.
Editors may, at their discretion, reject papers with a significant similarity report.If it is simply a matter of citing relevant texts or rephrasing some paragraphs, then resubmission can be encouraged.For a significant similarity report, the advantage of rejection with a request to resubmit is that there will be a new similarity check made on the resubmission.In the present system, this does not occur if the editor only asks for a revision.
If rejecting a paper on the grounds of quality at the initial decision phase, it is usually possible to do so positively.Quality is subjective but, by using the manuscript type descriptions, it should be possible to arrive at concrete reasons why the paper is inappropriate for GMD, and also to inform the authors of how a future submission of related work could be more successful.Bearing in mind the burden on reviewers it is important to refrain from sending very poorly written or seemingly unfinished work out for review.However, if in doubt, papers should be sent for peer review.Note that when rejecting as out of scope, it is now possible to specify other Copernicus/EGU journals which may be more appropriate.
If all the issues are dealt with adequately, the paper can be forwarded to the interactive public peer review.

C2 Reviewer call
After accepting a paper for interactive public peer review, the editor is required to call reviewers.
The usual number of reviewers for a GMD paper is two.It is expected that these researchers should be from different institutions, and that they should both be independent from the editor.Editors are encouraged to use internet resources to find appropriate reviewers outside their usual circle of collaborators.In the case of multi-component models spanning more than one sub-field of geoscience, more reviewers may be called to review different components of the model.GMD covers the whole field of geoscience, so it is not uncommon for editors to edit papers somewhat out of their main field of expertise.In such cases, it is recommended that there should be three reviewers.
In the case where a paper is well within the field of interest of the editor, they may choose to review the paper themselves but should only do so when two other reviewers are already in place or when multiple reviewer calls have failed to find a second reviewer.

C3 Interactive public peer-review period
The call to the reviewers goes out after the submitted manuscript is published online, after which the paper remains in open review for 8 weeks.During this time, the reviewers post their reviews which appear on the open-access GMD website for the paper and also fill in a checkpoint list indicating the quality of the paper.In addition, other scientists or members of the public may post comments, and the authors or editor may also add comments.
One idiosyncrasy with the system is that the second reviewer may read the first review and other comments before forming their own opinion, which can result in their review being largely a reaction to the discussion, rather than an independent view of the paper.This may be seen as either an advantage or disadvantage, but editors should be alert to the case of a purely reactive second review and at this point may call another reviewer, review the paper themselves or include additional comments in an editorial comment.
The review itself should focus on the clarity and rigour of the model description or development, and the extent to which it has been tested.It is not expected from a GMD paper that it contain novel scientific discoveries.If necessary, editors should be prepared to intervene and guide authors towards improved model description.The open-access format of the journal provides a useful way to issue that guidance in a constructive fashion.In particular, editorial comments published during the interactive discussion can be used to guide the discussion and indicate the progress through peer review.Editorial comments are particularly helpful at the start of the final response period and also when the final decision for acceptance or rejection for GMD is made.
Editors should make sure that someone (either a reviewer or the editor) has obtained the model or tool code.They may also attempt to compile the model, and run test cases where appropriate.In the case of other supplementary information, it should be downloaded and inspected with available software -e.g. the reviewers may open files in a NetCDF viewer.
In common with other journals, it is expected that editors will make their own comments on the manuscript and should guide the authors as to which parts of the reviews they should pay particular attention to.

C4 Closed peer-review period
After the first revision is submitted, the review process is no longer visible to the public in real time.When a paper is accepted for GMD, the subsequent reviews and editor's comments will be published alongside the final paper, although the reviewers and editors also have the option to submit private comments.The paper may continue through any number of revisions, with or without calls to the same or new reviewers.When the paper is finally accepted or rejected for GMD, the editor should write a comment to that effect on the open-access review page, explaining their decision, especially in the case of a manuscript which has received conflicting reviews.
Responsibility for the final decision on the manuscript rests with the assigned topical editor, but any difficulties may be referred to the executive editors.

C5 Special issues
GMD allows several papers to be collected together as a special issue.These can be used in the traditional sense to bring together a set of related papers over a defined time period.However, GMD special issues can be open-ended, and these allow papers describing incremental development of a model to be collected together indefinitely as the model develops.It is worth noting that due to their online nature, special issues can be created and added to retrospectively.It is also possible to create special issues in collaboration with other Copernicus/EGU journals.For example, this can be used to provide a home for MIP descriptions, with related model results published in other journals.
To propose a special issue, the authors should contact the Executive Editors in the first instance.The name of a contact person for the special issue is required.It is important to understand that manuscripts submitted to GMD special issues are handled the same way as other manuscripts: they are handled by the same topical editors (no "guest" editors for the special issue) and subject to the same editorial process.
Figure 1 shows the cumulative monthly submissions.The different manuscript types in GMD are expected to have different impacts on the community.Frequent citation of a full model description is a healthy sign, as it indicates that the model Published by Copernicus Publications on behalf of the European Geosciences Union.
the model development relates to a single model then the model name and the version number must be included in the title of the paper.If the main intention of an article is to make a general (i.e.model independent) statement about the usefulness of a new development, but the usefulness is shown with the help of one specific model, the model name and version number must be stated in the title.The title could have a form such as, "Title outlining amazing generic advance: a case study with Model XXX (version Y)".

Table 1 .
Evolution of impact factor through time.
N Pub : number of papers published in previous 2 years; N Cit : number of citations in year in column 1.The values in this table were calculated using the Thomson Reuters Web of Science database.