Error assessment of biogeochemical models by lower bound methods (NOMMA-1.0)

Sauerland, Volkmar; Löptien, Ulrike; Leonhard, Claudine; Oschlies, Andreas; Srivastav, Anand

doi:https://doi.org/10.5194/gmd-11-1181-2018

Articles | Volume 11, issue 3

https://doi.org/10.5194/gmd-11-1181-2018

© Author(s) 2018. This work is distributed under
the Creative Commons Attribution 3.0 License.

https://doi.org/10.5194/gmd-11-1181-2018

© Author(s) 2018. This work is distributed under
the Creative Commons Attribution 3.0 License.

Articles | Volume 11, issue 3

Methods for assessment of models

|

29 Mar 2018

Methods for assessment of models |

| 29 Mar 2018

Error assessment of biogeochemical models by lower bound methods (NOMMA-1.0)

Volkmar Sauerland, Ulrike Löptien, Claudine Leonhard, Andreas Oschlies, and Anand Srivastav

Download

Final revised paper (published on 29 Mar 2018)
Preprint (discussion started on 20 Jun 2017)

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

SC1: 'Executive Editor Comment on "Error assessment of biogeochemical models by lower bound methods"', Astrid Kerkweg, 14 Jul 2017
RC1: 'Review', Anonymous Referee #1, 24 Jul 2017
RC2: 'Error assessment of biogeochemical models by lower bound methods', Anonymous Referee #2, 31 Jul 2017
AC1: 'Reply to the executive editor', Volkmar Sauerland, 13 Sep 2017
AC2: 'Reply to referee 1', Volkmar Sauerland, 13 Sep 2017
AC3: 'Reply to referee 2', Volkmar Sauerland, 13 Sep 2017

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

AR by Volkmar Sauerland on behalf of the Authors (09 Oct 2017) Author's response Manuscript

ED: Referee Nomination & Report Request started (16 Oct 2017) by Christoph Müller

RR by Anonymous Referee #1 (02 Nov 2017)

Suggestions for revision or reasons for rejection

General Comments:
The authors significantly improved the Introduction and the Methods part, although some of the newly added text are not entirely clear (see below, detailed comments). My concern about the applicability of the proposed approach is not resolved. The authors significantly revised the relevant section 3.2, improving its readability, but preserved the previously presented example, which is too synthetic in terms of its simplicity and data density, failing therefore to make a convincing case (see below, the specific comments 1-2). Finally, my experience with the software was not entirely convincing either (specific comment 3). I still can not support publication of this work in GMD in its current form. I recommend the authors either to make a convincing case of the applicability of their approach with a non-synthetic data set with a more realistic data density, and displaying a realistic level of complexity. If this would be really beyond the scope of the intended work, authors may want to consider a more theoretically oriented journal specializing on statistical optimization.

Specific Comments:
1) Applicability of the approach in terms of complexity
The authors argue in P.17,L.3 that application of their approach to more complex systems is 'beyond the scope of this contribution'. I do not agree. For demonstrating that the approach is not only of theoretical interest, but intended to be of practical relevance to the target audience of GMD, an application to a realistically complex data set is needed.

2) Applicability of the approach in terms of data density
As an issue that seem to have emerging anew, the authors warn that their method require dense observations. Again, in reality, such dense observations (for the systems the authors claim their method is suitable for) almost never exist. So the applicability of the approach to a realistic observation set (e.g., with monthly or bi-weekly observations) is needed, with a thorough elaboration of the specific problems, and proposed solutions.

3) Software and its documentation
I downloaded and installed the code (NOMMA-1.0) and the prerequisite CPLEX software (IBM ILOG CPLEX Optimization Studio V12.7.0 for Linux), which went all fine by following the instructions in README files. The compiled 'regEx' binaries in both 'regression' and 'regressionCPX' folders seemed to run successfully, producing some .txt output. However, documentation of the software beyond installation seems to be entirely missing. In the absence of such documentation, my possibly incorrect understanding is that, the software is currently hardwired only to operate on a given data set, run a predefined set of tests with predefined parameters (such as the definition of number of extremes. If the code is provided for potential users, some documentation should describe how basic user options can be specified, or if the present code is provided only for exemplifying, an interpretation of examples should be provided.

Detailed Comments:
P5, L7-8. I don't see the logical link of this sentence to the previous text suggested by 'Therefore'.
P5, L17: 'intuitively clear': not to me really.

Hide

RR by Anonymous Referee #3 (17 Nov 2017)

Suggestions for revision or reasons for rejection

Review of manuscript „Error assessment of biogeochemical models by lower bound
methods (NOMMA-1.0)“ by Volkmar Sauerland, Ulrike Löptien, Claudine Leonhard, Andreas Oschlies, and Anand Srivastav

In their manuscript the authors address the important issue of an effective error assessment in biogeochemical modeling. The reliability of model simulations and future projections strongly depend on the models ability to reproduce observational data as good as possible. Therefore it is necessary to apply effective concepts of model parameter calibration techniques. Notably, a technique is required which ensures to end up in the global minimum of the parameter space when minimizing the root mean square error (RMSE).

In the present study the authors have developed the new concept of a “non-parametric optimization” which allows to calculate a lower bound of the RMSE by introducing a non-parametric function which fits the assemblage of observational data at best. This concept of a surrogate formulation of the optimization problem was successfully tested on synthetic and observed data (chlorophyll) from the Bornholm Basin in the Baltic Sea.

This is a valuable study addressing a new method of model parameter calibration. However, although the guidance by referee 1 led to a considerable improvement of the manuscript the Methods chapter 2 it is still hard to read for non specialists. As proposed by referee 1 I would like to see a schematic graphic illustration of the whole procedure for better understanding. I also miss a short paragraph addressing the applicability of the method to fully 3D ocean general circulation models.

One minor technical remark: At page 4 line 19 the brackets in the equation are missing.

Overall, save for the minor issues mentioned above, I can recommend publication in GMD.

Hide

ED: Reconsider after major revisions (21 Nov 2017) by Christoph Müller

AR by Volkmar Sauerland on behalf of the Authors (30 Jan 2018) Manuscript

ED: Referee Nomination & Report Request started (31 Jan 2018) by Christoph Müller

RR by Anonymous Referee #1 (13 Feb 2018)

RR by Anonymous Referee #3 (16 Feb 2018)

ED: Publish subject to technical corrections (17 Feb 2018) by Christoph Müller

AR by Volkmar Sauerland on behalf of the Authors (20 Feb 2018) Author's response Manuscript

Short summary

We present a concept to prove that a parametric model is well calibrated, i.e., that changes of its free parameters cannot lead to a much better model–data misfit anymore. The intention is motivated by the fact that calibrating global biogeochemical ocean models is important for assessment and inter-model comparison but computationally expensive.