Fldgen v1.0: an emulator with internal variability and space–time correlation for Earth system models

Link, Robert; Snyder, Abigail; Lynch, Cary; Hartin, Corinne; Kravitz, Ben; Bond-Lamberty, Ben

doi:https://doi.org/10.5194/gmd-12-1477-2019

Articles | Volume 12, issue 4

https://doi.org/10.5194/gmd-12-1477-2019

© Author(s) 2019. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/gmd-12-1477-2019

© Author(s) 2019. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 12, issue 4

Model description paper

|

12 Apr 2019

Model description paper |

| 12 Apr 2019

Fldgen v1.0: an emulator with internal variability and space–time correlation for Earth system models

Robert Link, Abigail Snyder, Cary Lynch, Corinne Hartin, Ben Kravitz, and Ben Bond-Lamberty

Download

Final revised paper (published on 12 Apr 2019)
Preprint (discussion started on 29 Mar 2018)

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

SC1: 'Executive Editor comment on "Computationally Efficient Emulators for Earth System Models"', Astrid Kerkweg, 02 May 2018
- AC1: 'Revised title for "Computationally Efficient Emulators for Earth System Models"', Robert Link, 23 May 2018
RC1: 'Response by anonymous reviewer to "Computing Efficient Emulators for Earth System Models".', Anonymous Referee #1, 05 Jun 2018
- AC2: 'Response to Anonymous Reviewer #1', Robert Link, 07 Sep 2018
RC2: 'Review of Link et al., Computationally Efficient Emulators for Earth System Models', Anonymous Referee #2, 13 Aug 2018
- AC3: 'Response to Anonymous Reviewer #2', Robert Link, 07 Sep 2018

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

AR by Robert Link on behalf of the Authors (12 Sep 2018) Author's response Manuscript

ED: Referee Nomination & Report Request started (27 Sep 2018) by Fiona O'Connor

RR by Anonymous Referee #2 (07 Oct 2018)

RR by Anonymous Referee #1 (12 Oct 2018)

Suggestions for revision or reasons for rejection

I recommend rejecting this paper for two reasons:

1. I am unhappy that the author has not decided to change 'emulator' to 'metamodel' or 'surrogate model' (the latter two words means the same thing). An emulator is a type of surrogate model that has particular properties. All three terms with their specific definitions have been used in the statistical / applied maths /scientific communities for at least 20 years. This is a big problem for me because if a key word (such as 'emulator') is defined differently by different papers in teh same journal (particularly GMD which has a wide and varied readership), it is very confusing for the readers of that journal (and the community as whole) as to what the actual meaning is. Tony O'Hagan defined an emulator in his 2006 paper (see section 2.2):
O’Hagan, A.: Bayesian analysis of computer code outputs: a tutorial, Reliability Engineering & System Safety, 91, 1290-1300, 2006

A sample of recent papers that have used emulators (using this well established definition) are:
(i) Lee, L., Pringle, K., Reddington, C., Mann, G., Stier, P., Spracklen, D., Pierce, J., and Carslaw, K.: The magnitude and causes of uncertainty in global model simulations of cloud condensation nuclei, Atmos. Chem. Phys, 13, 8879-8914, 2013.
(ii) Chang, E. T., Strong, M., and Clayton, R. H.: Bayesian sensitivity analysis of a cardiac cell model using a Gaussian process emulator, PloS one, 10, e0130252, 2015.
(iii) Efficient Emulation of Radiative Transfer Codes Using Gaussian Processes and Application to Land Surface Parameter Inferences, Remote Sensing, 8, 119, 2016.
(iv) Kennedy, M., Anderson, C., O'Hagan, A., Lomas, M., Woodward, I., Gosling, J. P., and Heinemeyer, A.: Quantifying uncertainty in the biospheric carbon flux for England and Wales, Journal of the Royal Statistical Society: Series A (Statistics in Society), 171, 109-135, 2008.
(v) Williamson, D., Goldstein, M., Allison, L., Blaker, A., Challenor, P., Jackson, L., & Yamazaki, K. (2013). History matching for exploring and reducing climate model parameter space using observations and a large perturbed physics ensemble. Climate dynamics, 41(7-8), 1703-1729.

2. I just don't understand what this paper is about. Or at least I think I understand what the authors are trying to do, but the communication of this methodology is way too technical for a journal like GMD which has a readership with a wide and diverse backgrounds. Most of the technical aspects use statistics and mathematics. If I as a statistician and mathematican cannot follow what's going, how do the authors expect a non-statistiician (e.g. the atmospheric chemistry modellers I work with) to understand this paper.

I actually think that this paper should be submitted as two papers:
The first could set out the technical statistical and mathematical methodology in a statistical journal such as 'Uncertainty Quantification'. The second could then be a more applied paper for the scientific community

Hide

RR by Anonymous Referee #3 (05 Feb 2019)

Suggestions for revision or reasons for rejection

This presents a useful and novel approach to ESM emulation, capturing internal variability that is generally neglected but is important for many impact evaluations. I am happy with the responses to the first round of reviews, with the exception of the final point below, which I believe should be discussed in the manuscript, and not confined to the review response.

The title is not very informative - all emulators are computationally efficient, so what marks this one out? A title that refers to internal variability might be useful, and would help to draw the attention of target end-users?

Specific examples of cases when impacts are strongly dependent upon internal variability might be useful to make it crystal clear what this is doing and why it is needed. Even just pulling in the headline result from Ray “Climate variation explains a third of global crop yield variability” would help non-specialist readers a lot I think.

I had similar thoughts as reviewer two re using different RCPs used to train the emulator, and I didn’t find the response fully convincing – some tests should be possible here? In particular, I found myself wondering whether EOF1 reflected responses to different RCPs. Did you check whether EOF1 was similar in an emulator trained on a few simulations with a single RCP (and didn’t disappear, as when you trained on a single simulation)? Why not compare the variance of the full emulator (i.e. trained on all 9 simulations) separately with the RCP2.6 simulations and with the RCP8.5 simulations to quantify any scenario bias? In any event, some discussion of this should be included in the text as I suspect many people will question it. Repeating from Reviewer Two “Section 4.2 got me thinking that as the model is trained on the RCP outputs, is there any difference in the results when taking just the set of realisations from RCP2.6 and RCP8.5? Certaintly across ESMs, the variance across models increases with increasing global mean temperature. It would therefore not be correct to use a variability model that is trained on RCP8.5 for low forcing scenarios or those with a peak and decline. I note the authors address this in section 4.3, but I wonder if they have tested this.”

Hide

ED: Publish subject to technical corrections (28 Feb 2019) by Fiona O'Connor

ED: Publish subject to minor revisions (review by editor) (01 Mar 2019) by Fiona O'Connor

AR by Robert Link on behalf of the Authors (06 Mar 2019) Author's response Manuscript

ED: Publish as is (26 Mar 2019) by Fiona O'Connor

AR by Robert Link on behalf of the Authors (27 Mar 2019)

Short summary

Earth system models (ESMs) produce the highest-quality future climate data available, but they are costly to run, so only a few runs from each model are publicly available. What is needed are emulators that tell us what would have happened, if we had been able to perform as many ESM runs as we might have liked. Much of the existing work on emulators has focused on deterministic projections of average values. Here we present a way to imbue emulators with the variability seen in ESM runs.