Inconsistent strategies to spin up models in CMIP5: implications for  ocean biogeochemical model performance assessment

Séférian, Roland; Gehlen, Marion; Bopp, Laurent; Resplandy, Laure; Orr, James C.; Marti, Olivier; Dunne, John P.; Christian, James R.; Doney, Scott C.; Ilyina, Tatiana; Lindsay, Keith; Halloran, Paul R.; Heinze, Christoph; Segschneider, Joachim; Tjiputra, Jerry; Aumont, Olivier; Romanou, Anastasia

doi:https://doi.org/10.5194/gmd-9-1827-2016

Articles | Volume 9, issue 5

https://doi.org/10.5194/gmd-9-1827-2016

© Author(s) 2016. This work is distributed under
the Creative Commons Attribution 3.0 License.

https://doi.org/10.5194/gmd-9-1827-2016

© Author(s) 2016. This work is distributed under
the Creative Commons Attribution 3.0 License.

Articles | Volume 9, issue 5

Methods for assessment of models

|

12 May 2016

Methods for assessment of models |

| 12 May 2016

Inconsistent strategies to spin up models in CMIP5: implications for ocean biogeochemical model performance assessment

Roland Séférian, Marion Gehlen, Laurent Bopp, Laure Resplandy, James C. Orr, Olivier Marti, John P. Dunne, James R. Christian, Scott C. Doney, Tatiana Ilyina, Keith Lindsay, Paul R. Halloran, Christoph Heinze, Joachim Segschneider, Jerry Tjiputra, Olivier Aumont, and Anastasia Romanou

Download

Final revised paper (published on 12 May 2016)
Supplement to the final revised paper
Preprint (discussion started on 13 Oct 2015)

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

RC C2795: 'Review', Anonymous Referee #1, 05 Nov 2015
EC C2971: 'Uber-minor comment on Figure 1', Andrew Yool, 19 Nov 2015
RC C3026: 'Important paper on model spin up and model evaluation/ranking', Fortunat Joos, 23 Nov 2015
RC C3448: 'Interactive comment on “Inconsistent strategies to spin up models in CMIP5: implications for ocean biogeochemical model performance assessment” by R. Séférian et al.', Iris Kriest, 22 Dec 2015
AC C3891: 'Response to reviewers', Roland Séférian, 29 Jan 2016

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

AR by Roland Séférian on behalf of the Authors (29 Jan 2016) Author's response Manuscript

ED: Referee Nomination & Report Request started (10 Mar 2016) by Andrew Yool

RR by Fortunat Joos (14 Mar 2016)

RR by Anonymous Referee #1 (18 Mar 2016)

Suggestions for revision or reasons for rejection

Inconsistent strategies to spin up models in CMIP5: implications for ocean biogeochemical model performance assessment

Unfortunately after a further reading of the manuscript I still think there is much work to be done. In general, the language is often rather impenetrable or imprecise. I have provided a few examples below, but I feel that the manuscript as a whole should be carefully re-read by the team of co-authors (I suspect this hasn’t happened yet). Also, unless Im missing something I find many aspects of the analysis to be inappropriate.

General comment on figures. I don’t know if this is the policy of the journal but its extremely frustrating not to have figure captions next to the figures, The reader has to be constantly jumping around the document. Also all figures should have panel labels added.

Specific Comments:
In the comments below I refer to line numbers from the track changed pdf: gmd-2015-194-author_response-version2.pdf

94-97. Fig 1 should be referred to. You should also acknowledge other initialisation protocols (e.g. use of previous model runs)

100: Recent work suggests that these models will not reach a steady state as there are energy leaks in the system: http://journals.ametsoc.org/doi/abs/10.1175/JCLI-D-15-0477.1
Im not sure if similar issues exist for BGC components, but if the physical system has a perpetual drift this is also going to affect the BGC. This puts into question the idea of an exponentially decreasing drift.

108-110: I disagree, there are plenty of studies that do drift correction (including the analysis of ocean variables done by the IPCC) - so an equilibrium is not assumed it is corrected for.

127-133: Sentence needs splitting for readability

133: The latter SET OF VARIABLES are initialized …

163-165: deep is a subset of subsurface, so this doesnt really make sense

167-170: Not clear what is meant here. If there is a continuous flux of carbon into the ocean then how can there be an equilibrium?
Do you mean: … ~10,000 years to reach a state where a global sea-air carbon flux is less than 0.01 Pg C y-1 .

181-182: I dont follow the second point. Why does spin up duration depend on knowing the preindustrial carbon stored?

372: do you mean 'computed at different depths'?

380-382: Your choice of what constitutes drift appears to be purely subjective. Why do you think that variability below 100yrs is internally driven while variability above 200yrs is associated with drift?

425: total difference (RMSE)
Why not just call it RMSE, why introduce an ambiguous term like ‘total difference’

425-426: an RMSE error gives the typical error at a location. Not the error in the global average

447: … drifts reported for CMIP5 ESMs.
This requires a reference

457: To assess the global sea-to-air carbon flux …
Do you mean: ‘To estimate the observed pre-industrial global carbon flux ...’?

451-453: Do you mean that this metric has been used to assess if models are equilibrated?

471-472 it is weaker…
I suspect that this is wishful thinking. Is there a significant difference between the two trends?

467-478. I feel that this section needs to be much more clearly explained. I Don’t understand what you are doing here. A number of issues:
-I suspect that the statement that the linear trend over the last 100 years is smaller than the 250-500 trend is wishful thinking. Is there a significant difference between the two trends?
-Im not clear how you are constructing the exponential model. Do you calculate a sequence of 100yr trends from all the data and then fit your exponential as you do later in the paper?
-I don’t really understand what the 0.4-0.56 number represents. Are you saying that there was a pre-industrial outgassing of this amount? Given model biases why would we expect model equilibrium to be the same as the observed equilibrium?
-If I understand correctly, you use a linear trend to calculate an intersection between the trend and the 0.4-0.56 range. So you are assuming that the drift magnitude now stays the same over the next 1100-1300 years. You then use an exponential model to estimate the rate of drift after 1100-1300 years (that was obtained using a linear drift model). Either I have misunderstood this section, or your method doesn’t seem to make sense.

482: couldnt this equally lead to an overestimate of the time?

Figure 3 caption: what are the [X]’s

495-496: But according to your figure Alk-DIC lies within the observational range of uncertainty.

522: The panel labels in Figure 4 are not shown. Please check this for all figures. Also where appropriate panel and figure should be provided in the text (not just figure number)

538: …pattern of errors are well correlated.
well correlated with what? Do you mean there is a high level of spatial autocorrelation? If so, what is the implication of this?

538-539: It directly translates the assumptions employed in the biogeochemical model
what does this mean?

The descriptions of the figures 4 5 & 6 are largely the same. What insight is gained from the changes at different depths.
Unless something interesting can be concluded I suggest deleting 1 or 2 of these figures

545: total mismatch
It would be very helpful if precise terminology was used (this applies at many places in the manuscript). What does ‘total mismatch’ mean? I presume its the evolution of globally averaged rmse at different depth levels. The figure caption isnt clear either.

551-552: after few decades within the upper hundred meters …
But in figure 3 you show that it takes O[250yrs] to reach a quasi steady state.

555: respectively in relation with the structure of the large-scale ocean circulation.
I don’t understand what this means.

556-557: RMSE evolves much slower because this depth corresponds to the depth of the very old radiocarbon age
this statement of causality makes no sense. I presume you are trying to say that ventillation is slow in these regions as evidenced by large radiocarbon age.

559-560: Why?

564-565: equilibration after a longer of …
There’s something wrong with this sentence

568: is fitted level to the 80 drift values for
what does 'is fitted level' mean?

571: The simple drift model fits well the evolution of the drift …
Based on your definition of drift i.e. 100 yr linear trend, the exponential model looks very different to the data you are trying to fit to. You are not getting any of the centennial variability (that by your definition constitutes variability in the drift).
In general, I am not clear why you have taken this approach. If you think that drift follows an exponential model why not just fit an exponential to the raw RMSE data without first calculating linear trends.
Also in figure 8 caption you say that the magenta line is: 'The best-fit linear regressions'. This cannot be correct as the lines aren’t linear.

573: Unless Im misunderstanding what you are doing this comparison in term of correlation is not meaningful. What are you trying to test? that the slopes are same? (for this you wouldnt look at correlation) or that the variability is coherent? (you this you would use correlation but the model doesnt have any variability). Moreover the data certainly does not have 80 degrees of freedom -there is massive temporal auto correlations. The fitted model would have far less.

Figure 9 caption. As in figure 8, the green line is not a linear fit. This is presumably the fit to the exponential model

608: this relationship suggests a general decrease of the drift as a function of spin-up duration
For 150m it looks like the CI would suggest that the fit is not different from a slope of zero i.e. no relationship. You also contradict yourself below when you say: ‘at 150 m depth and hence indicating that there is no link’
Similar to my comments for figure 8, I don’t believe this correlation analysis is meaningful. 1. Correlation is a test for a linear relationship 2. Its not used to assess the relationship between discrete data and a line of best fit. A fit line would have very few degrees of freedom, so you couldnt assume 15 independent samples.

615: This low significance level
do you mean low correlation? your terminology here and elsewhere needs checking

620-623:
Is this an extremely complicated way of saying that in IPSL the drift also reduces with time?

625-627: If the spin up is too short to determine robust drifts, doesnt this invalidate your whole section regarding this model?

639: why do you call it distance. I presume you mean the 'ensemble mean globally averaged RMSE'. As mentioned above it would be really helpful if you stick to precise terminology

Figure 10 x-axis should have units

697: Just because circulation reaches a steady state doesnt mean that other important physical variables like temperature arent still drifting

701: I dont understand what you mean by error propagation. You havent mentioned this in the results.

734-756. This point is illustrated …
I don’t see how this illustrates the above point. Im lso not clear what the above point means.

745-748. Sentence doesn’t make sense to me.

753-756. Again this doesn’t make sense to me. Spread in what? In fact Im struggling to understand what this whole paragraph is trying to say.

826-829. Im not sure that an exponential model is most appropriate here in many global cases (let alone regional cases) (see your fig 8)

831: The above-mentioned remark
Which remark?

831: the relatively low confidence level of the fit
This terminology is very awkward. Do you mean the poor fit of the exponential model to the drift data? In one sentence you talk about confidence in the next you talk about significance.

840-841: it is unlikely that model fields drift at the same rate along the spin-up simulation
Isnt this why you use an exponential model which changes over time?

859: 3-dimensional growth rate
What is this?

Hide

RR by Iris Kriest (27 Mar 2016)

ED: Publish subject to minor revisions (Editor review) (01 Apr 2016) by Andrew Yool

AR by Roland Séférian on behalf of the Authors (19 Apr 2016) Author's response Manuscript

ED: Publish as is (22 Apr 2016) by Andrew Yool

AR by Roland Séférian on behalf of the Authors (22 Apr 2016)

Short summary

This paper explores how the large diversity in spin-up protocols used for ocean biogeochemistry in CMIP5 models contributed to inter-model differences in modeled fields. We show that a link between spin-up duration and skill-score metrics emerges from both individual IPSL-CM5A-LR's results and an ensemble of CMIP5 models. Our study suggests that differences in spin-up protocols constitute a source of inter-model uncertainty which would require more attention in future intercomparison exercises.