An alternative way to evaluate chemistry-transport model variability

Menut, Laurent; Mailler, Sylvain; Bessagnet, Bertrand; Siour, Guillaume; Colette, Augustin; Couvidat, Florian; Meleux, Frédérik

doi:https://doi.org/10.5194/gmd-10-1199-2017

Articles | Volume 10, issue 3

https://doi.org/10.5194/gmd-10-1199-2017

© Author(s) 2017. This work is distributed under
the Creative Commons Attribution 3.0 License.

https://doi.org/10.5194/gmd-10-1199-2017

© Author(s) 2017. This work is distributed under
the Creative Commons Attribution 3.0 License.

Articles | Volume 10, issue 3

Methods for assessment of models

|

17 Mar 2017

Methods for assessment of models |

| 17 Mar 2017

An alternative way to evaluate chemistry-transport model variability

Laurent Menut, Sylvain Mailler, Bertrand Bessagnet, Guillaume Siour, Augustin Colette, Florian Couvidat, and Frédérik Meleux

Download

Final revised paper (published on 17 Mar 2017)
Preprint (discussion started on 24 Jun 2016)

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

RC1: 'Referee report, Menut et al', Anonymous Referee #1, 03 Aug 2016
RC2: 'Review of publication “An unusual way to validate regional chemistry-transport models” by L Menut et al.', Anonymous Referee #2, 19 Oct 2016
AC1: 'Answers to reviewers 1 and 2', Laurent Menut, 19 Dec 2016

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

AR by Laurent Menut on behalf of the Authors (19 Dec 2016) Manuscript

ED: Referee Nomination & Report Request started (20 Dec 2016) by Slimane Bekki

RR by Anonymous Referee #2 (23 Jan 2017)

Suggestions for revision or reasons for rejection

Second review of: An unusual way to validate regional chemistry-transport models” by Lauren Menut et al.

The methodology proposed by the authors is original and has potential to complement the traditional approach. Unfortunately, as presented in this publication, the added value of this work remains limited because subjective judgements must be made to interpret the results. I also feel that my previous comments have been accounted for, only very partially.

1. The bibliography has been extended but those references are not used much in the text. For example the decomposition of the main indicator RMSE (or MSE) into its three components (Solazzo and Galmarini, Thunis et al., Taylor et al.) could be used as starting point to identify the different indicators and justify the choice of the correlation as central one for this study. The decomposition into a systematic and unsystematic error has already been done in other works that are not referenced.
2. Even if not applicable in a diagram, information on the bias or on the standard deviation could be provided. Values of SN and D could be calculated and added to the analysis. The fact that the bias is low and does not show any variability (e.g. for T2m) is an interesting information per se. It could also be that the bias shows more variability for other species and then become the crucial parameter to analyse.
3. My point about the use of the “score” terminology has not been addressed. This term is used (55 times in the whole document) indifferently to describe the indicator (e.g. p2 l5-6, p3 l63, p3 l87) and the value taken by this indicator (p4 l55), which makes it confusing to understand. Could the Author give a definition of “score” and check that it is consistent though the document?
4. The title does not reflect what the methodology is about. I agree with the Authors that the methodology is unusual but many things can be defined as unusual. I believe that the title should provide insight on the novel aspects discussed in the document (the use of different meteorological years)
5. Regarding the qualitative aspects of the methodology. I agree that the indicators are calculated quantitatively but the judgement made on whether the results are good or bad remains subjective. This judgement is based on expert knowledge (e.g. that a correlation of 0.5 is very good for a given species). In my view, this limits the benefit of the methodology, as users need to know a-priori what a good behaviour is. I understand that the key point is in the use of several years of data but if at the end the interpretation of the indicator depends on expert judgement, this is a limitation. Examples (p5 l25-27; p5 l47-50; p6 l11)
6. I still do not understand why observations cannot be used to fix a minimum threshold. According to me, values of SN and D calculated on the only basis of the set of observations (substituting the model value by the observation of the reference year) could be calculated to make the approach a little bit less subjective.

7. English has been improved but many misspells and unclear sentences remain (only few examples provided below).

Minor comments

8. P2 l32 require
9. P2 l64 A couple of lines would be needed to indicate that the Authors now start the description of the methodological approach.
10. P3 l66-69: unclear
11. P3 l72: The bias is an indicator, not a score!
12. P3 l73: I disagree with the Authors, the RMSE is not driven by bias. Depending on the variable and the period of time considered, the RMSE can be dominated by correlation, bias or by standard deviation.
13. P3 l78: unclear formulation, please re-phrase.
14. Figure 1: I guess MYV should be Imv
15. p5 l 23: have --> has
16. p5: why a subscript “s” in Ds
17. Figure 2; MYV should become Imv I guess
18. P5 l70: unclear, please re-phrase
19. P6 l14: year --> years
20. P6 l16-18: please re-phrase
21. P6 l27-28: disagree: see point 12

Hide

RR by Anonymous Referee #1 (05 Feb 2017)

ED: Reconsider after major revisions (12 Feb 2017) by Slimane Bekki

AR by Laurent Menut on behalf of the Authors (18 Feb 2017) Manuscript

ED: Publish subject to technical corrections (23 Feb 2017) by Slimane Bekki

AR by Laurent Menut on behalf of the Authors (01 Mar 2017) Author's response Manuscript

Short summary

A simple and complementary model evaluation technique for regional chemistry transport is discussed. The methodology is based on the concept that we can learn about model performance by comparing the simulation results with observational data available for time periods other than the period originally targeted.