the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 13 Jul 2021
Systematic global evaluation of accuracy of seasonal climateforecasts for monthly precipitation of JMA/MRI-CPS2 bycomparing with a statistical system using climate indices
Abstract. In this study, we aimed to evaluate the monthly precipitation forecasts of JMA/MRI-CPS2, a global dynamical seasonal climate forecast (Dyn-SCF) system operated in the Japan Meteorological Agency, by comparing them with the forecasts of a statistical SCF (St-SCF) system using climate indices systematically and globally. Accordingly, we developed a new global St-SCF system using 18 climate indices and compared the monthly precipitation of this system with those of JMA/MRI-CPS2. Consequently, it was found that JMA/MRI-CPS2 forecasts are superior to St-SCFs around the equator (10° S–10° N) even for six-month lead forecasts. For one-month lead forecasts, the accuracy of JMA/MRI-CPS2 forecasts was higher than that of St-SCFs when viewed globally. In contrast, for forecasts made two months or longer in advance, St-SCFs had an advantage in global forecasts. In addition to evaluating the accuracy of JMA/MRI-CPS2 forecasts, the slow dynamics of the ocean and atmosphere, not reproduced by the JMA/MRI-CPS2 system, were determined by comparing the evaluations, and it was concluded that this could contribute to improving Dyn-SCF systems.
- Preprint
(5646 KB) - Metadata XML
- BibTeX
- EndNote
Status: closed
-
RC1: 'Comment on gmd-2021-131', Anonymous Referee #1, 11 Oct 2021
- AC1: 'Reply on RC1', Yuji Masutomi, 10 Dec 2021
- AC2: 'Reply on RC1', Yuji Masutomi, 10 Dec 2021
-
AC4: 'Reply on RC1', Yuji Masutomi, 10 Dec 2021
Please ignore AC1, because I uploaded a wrong file.
Citation: https://doi.org/10.5194/gmd-2021-131-AC4
-
RC2: 'Comment on gmd-2021-131', Anonymous Referee #2, 18 Oct 2021
This study sets out to compare global precipitation forecasts generated by a dynamical SCF and a climate-index based statistical forecasting system. It is found the dynamical forecasts performed better in the first month and the statistical forecasts performed better for longer lead times, with some nuances over land/ocean and around the equator. It is a potentially interesting study and the paper is presented well. However, I believe the study suffers a major flaw that the authors will need to rectify, and some additional work that pushes the revisions beyond major. Here are my major and minor comments:
Major comments
- The statistical forecasting methodology adopts the predictor with the highest correlation after model fitting (Figure 1 and lines 82-88 ). This means the selected climate indices are based on peeking during the evaluation period and selected to maximise the performance over the evaluation period. With 18 climate indices, the risk of artificially inflating the performance of the statistical system is high. I suggest that the selected climate index for each year ought to vary based on total exclusion of the data for that year through all steps up to and including the calculation of the correlation.
- Seasonal forecasting is highly uncertain, and the deterministic forecasts of the type produced here may be of limited value to end users. I suggest that the authors consider a means to produce uncertainty estimates or "ensembles" and include some more typical SCF analysis and verification results. I believe that is important to analyse the underlying skill in the ensemble mean as the authors have done, however, adopting it as the only verification approach side-steps a major advantage of the dynamical forecasts in having ensembles.
Minor comments:
- I suggest that the motivation for studying precipitation should be established earlier in the introduction, and a comment that statistical forecasts are suited to forecasting a small number of variables.
- The process of selecting a different climate index for each cell could lead to very spatially inconsistent forecasts. I suggest this is addressed in the revised discussion.
- It is one intention of this study to highlight that dynamical models don’t always capture the slow dynamics and teleconnections. However, the paper is rather brief and doesn’t attempt to explain why particular climate indices are important, or even the frequency at which they are selected. I suggest in the revision some further attention is given to highlighting the most important indices.
- Mean square error and skill scores are referred to in the introduction, however they are not used in the paper. I suggest making the introduction and methods more consistent.
- It sounds to me like the climate indices wouldn't be available in time to make real time forecasts due the week delay (e.g. section 4.3). I suggest addressing this issue in the revision.
Citation: https://doi.org/10.5194/gmd-2021-131-RC2 - AC3: 'Reply on RC2', Yuji Masutomi, 10 Dec 2021
Status: closed
-
RC1: 'Comment on gmd-2021-131', Anonymous Referee #1, 11 Oct 2021
- AC1: 'Reply on RC1', Yuji Masutomi, 10 Dec 2021
- AC2: 'Reply on RC1', Yuji Masutomi, 10 Dec 2021
-
AC4: 'Reply on RC1', Yuji Masutomi, 10 Dec 2021
Please ignore AC1, because I uploaded a wrong file.
Citation: https://doi.org/10.5194/gmd-2021-131-AC4
-
RC2: 'Comment on gmd-2021-131', Anonymous Referee #2, 18 Oct 2021
This study sets out to compare global precipitation forecasts generated by a dynamical SCF and a climate-index based statistical forecasting system. It is found the dynamical forecasts performed better in the first month and the statistical forecasts performed better for longer lead times, with some nuances over land/ocean and around the equator. It is a potentially interesting study and the paper is presented well. However, I believe the study suffers a major flaw that the authors will need to rectify, and some additional work that pushes the revisions beyond major. Here are my major and minor comments:
Major comments
- The statistical forecasting methodology adopts the predictor with the highest correlation after model fitting (Figure 1 and lines 82-88 ). This means the selected climate indices are based on peeking during the evaluation period and selected to maximise the performance over the evaluation period. With 18 climate indices, the risk of artificially inflating the performance of the statistical system is high. I suggest that the selected climate index for each year ought to vary based on total exclusion of the data for that year through all steps up to and including the calculation of the correlation.
- Seasonal forecasting is highly uncertain, and the deterministic forecasts of the type produced here may be of limited value to end users. I suggest that the authors consider a means to produce uncertainty estimates or "ensembles" and include some more typical SCF analysis and verification results. I believe that is important to analyse the underlying skill in the ensemble mean as the authors have done, however, adopting it as the only verification approach side-steps a major advantage of the dynamical forecasts in having ensembles.
Minor comments:
- I suggest that the motivation for studying precipitation should be established earlier in the introduction, and a comment that statistical forecasts are suited to forecasting a small number of variables.
- The process of selecting a different climate index for each cell could lead to very spatially inconsistent forecasts. I suggest this is addressed in the revised discussion.
- It is one intention of this study to highlight that dynamical models don’t always capture the slow dynamics and teleconnections. However, the paper is rather brief and doesn’t attempt to explain why particular climate indices are important, or even the frequency at which they are selected. I suggest in the revision some further attention is given to highlighting the most important indices.
- Mean square error and skill scores are referred to in the introduction, however they are not used in the paper. I suggest making the introduction and methods more consistent.
- It sounds to me like the climate indices wouldn't be available in time to make real time forecasts due the week delay (e.g. section 4.3). I suggest addressing this issue in the revision.
Citation: https://doi.org/10.5194/gmd-2021-131-RC2 - AC3: 'Reply on RC2', Yuji Masutomi, 10 Dec 2021
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 1,248 | 407 | 77 | 1,732 | 64 | 67 |
- HTML: 1,248
- PDF: 407
- XML: 77
- Total: 1,732
- BibTeX: 64
- EndNote: 67
Viewed (geographical distribution)
| Country | # | Views | % |
|---|---|---|---|
| United States of America | 1 | 435 | 26 |
| China | 2 | 352 | 21 |
| Germany | 3 | 178 | 10 |
| Japan | 4 | 162 | 9 |
| Netherlands | 5 | 62 | 3 |
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
- 435
