the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Submitted as: model experiment description paper 11 Oct 2021
Submitted as: model experiment description paper | 11 Oct 2021
Prediction Error Growth in a more Realistic Atmospheric Toy Model with Three Spatiotemporal Scales
- 1Max Planck Institute for the Physics of Complex Systems (MPIPKS), D-01187, Dresden, Germany
- 2Department of Atmospheric Physics, Faculty of Mathematics and Physics, Charles University, 18000, Prague, Czech Republic
- 1Max Planck Institute for the Physics of Complex Systems (MPIPKS), D-01187, Dresden, Germany
- 2Department of Atmospheric Physics, Faculty of Mathematics and Physics, Charles University, 18000, Prague, Czech Republic
Abstract. This article studies the growth of the prediction error over lead time in a schematic model of atmospheric transport. Inspired by the Lorenz (2005) system, we mimic an atmospheric variable in 1 dimension, which can be decomposed into three spatiotemporal scales. We identify parameter values that provide spatiotemporal scaling and chaotic behavior. Instead of exponential growth of the forecast error over time, we observe a more complex behavior. We test a power law and the quadratic hypothesis for the scale dependent error growth. The power law is valid for the first days of the growth, and with an included saturation effect, we extend its validity to the entire period of growth. The theory explaining the parameters of the power law is confirmed. Although the quadratic hypothesis cannot be completely rejected and could serve as a first guess, the hypothesis’s parameters are not theoretically justifiable. In addition, we study the initial error growth for the ECMWF forecast system (500 hPa geopotential height) over the 1986 to 2011 period. For these data, it is impossible to assess which of the error growth descriptions is more appropriate, but the extended power law, which is theoretically substantiated and valid for the Lorenz system, provides an excellent fit to the average initial error growth of the ECMWF forecast system. Fitting the parameters, we conclude that there is an intrinsic limit of predictability after 22 days.
Hynek Bednář and Holger Kantz
Status: final response (author comments only)
-
RC1: 'Comment on gmd-2021-256', Anonymous Referee #1, 02 Nov 2021
The topics addressed in this manuscript are very interesting since the study of the error growth in multi-scale nonlinear systems is fundamental to geophysical modelling, at various levels of complexity. The authors present a large set of results obtained from simplified multi-scale atmospheric models and discuss their relevance in comparison with ECMWF forecasting system data.
I think this manuscript needs some minor revision before publication. My remarks follow below.
1) I wonder why the authors did not carry out their analysis (i.e. error growth and error gorwth rate in nonlinear multi-scale systems) by adopting a scale-dependent point of view, i.e. measuring the scale-dependent error growth rate by means of the so-called Finite-Scale (or Finite-Size) Lyapunov Exponent (FSLE), a quantity that was introduced and developed specifically for this kind of studies, see E. Aurell et al., Phys Rev Lett. 1996, 77(7), 1262-1265; E. Aurell et al., 1997, J. Phys. A: Math. Gen. 30 1. The time-dependent description is not necessarily wrong or inaccurate but, in general, can be affected by "scale-interference" issues that do not help the investigation of power laws and scaling exponents, see also, at this regard, the discussion reported in Boffetta et al., 2000, Chaos 10 (1), pp 50-60.
2) lines 33-34: Why a scale-dependent error growth (small errors growing faster than large errors) should imply that the true Lyapunov exponent of the system is infinite ? The scale.dependent error growth rate actually is expected to converge to a constant value in the limit of infinitesimal errors, and that limit is assumed as the true maximum Lyaunov exponent of the system. The benefit of using a scale-dependent description of the error growth is just to have information on the error growth over all the observable scale range, regardless of the actual value of the maximum Lyapunov exponent, which is not very relevant if the smallest scales cannot be resolved.
3) lines 34-35: In a paper dealing with scale-dependent prediction errors, I think the FSLE-related bibliography cannot be reduced to only one paper (Cencini and Vulpiani, 2013). See point 1) for some more references.
4) line 225: In the formula of the Max Lyap Exp there should be an epsilon (i,e. the infinitesimal initial error size) in the denominator.
5) lines 403-404: In the linit of small scale errors, the growth is expected to be exponential, if the system is chaotic, so a power law is necessarily an approximation of the true behavior. If a power law looks like a reasonable approximation of small scale error growth, does this imply that the error size is significantly above the smallest scale limit ?
6) lines 416-424: it seems that, at least as far as ECMWF forecasting system is concerned, the authors' conjecture, although succesfully tested on their simplified 3-scale model, does not improve significantly the description of the error growth evolution with respect to other fitting functions.
In conclusion, I would like to stress once again that some points could be better clarified and the whole manuscript could be substantially improved if the authors cosidered checking their results with a FSLE-based error gowth analysis. Notice, apparently the quantity defined as error growth rate by the authors might look equivalent to the FSLE but there is a fundamental difference: the FSLE is an intrinsic scale-dependent indicator, i.e. it is measured by taking the average of the growth time at fixed scale, while the error growth rate reported in the manuscript, if I got it right, comes always from a fixed-time average. In case they were exactly the same quantity this should be clearly specified in the text.
-
AC1: 'Reply on RC1', Hynek Bednar, 21 Dec 2021
We are grateful to the referee for devoting their time to our manuscript. The valuable comments and suggestions will help us to improve the paper.
We will here respond to the main comments made:
1) We are very sorry for not citing all of the relevant literature on scale dependent error growth, in particular in turbulence, and how to measure it numerically. We will fix this in the revised version.
Concerning the calculation of the error growth rates, we were indeed using a different scheme than proposed in Aurell et al PRL 77 1262 (1996): We create a perturbed field close to the reference field, we iterate them both for some transient time to allow the perturbation to re-direct into the locally most unstable direction, and then track the increase of the perturbation in time. Then (Fig.2a)
we average over the error growth rate at given times after initialization, and plot it versus the mean error magnitude at that time (called E(t) in Fig.2a). The reason for doing so is that in real forecasts, it is standard to study the average error after some given time. In particular, when performing the analysis for the ECMWF ensemble forecasts, we are only evaluating error growth experiments which have been performed by others, so that we do only have access to the error growth rate after fixed times, and not at given error magnitudes. We fully agree that the way how the averages are obtained generally will have an influence on the numerical results. We have therefore now repeated the error
growth analysis for our model system in the way of Aurell et al. Qualitatively, the results are the same. With our method, the power law behavior λ(E) ∝ E−σis slightly better than for Aurell’s method, which seems to have the tendency to resolve the Lyapunov exponents of the different levels. This smoothing of the error growth rate most certainly comes from the fact that, as suspected by the referee, our method mixes the scales: In the average over many error growth experiments, at some given time t after initialization, the actual errors E(t) have different magnitudes. In addition, we observe a slightly lower error growth rate for the same value of error magnitude, which we understand also by this mixing of scales: If λ(E) ∝ E−σ, then ‹λ(E)› < ‹E›−σ, where the average is assumed to be done over the error magnitudes which are found after fixed time t.So our conclusion is that although a scale dependent error growth per se should be studied in the way of Boffetta et al, a comparison with real weather forecast data can only be done by studying the time evolution of errors. We will add a corresponding discussion in the revised version of the manuscript.
2) line 33-34: Our statements about the “true” largest Lyapunov exponent being infinite were imprecise. They refer to a model where the scale dependent Lyapunov exponent reads λ(ε) ∝ ε−|σ| and hence λ → ∞ for ε → 0. Evidently, such a behavior would be an idealization (approximation) since in real systems like in turbulence one would expect to have some cut-off at some small length scales with a cross-over to some finite λmax. A comment like this was made in the publication Brisch & Kantz to which we refer in the manuscript, but I agree that we have to include it here as well.
3) We are sorry for ignoring relevant literature on scale dependent error growth and will give adequate merits to that in the revised version.
4) line 225: We fully agree, we erroneously forgot a prefactor 1/ ln ε. We are grateful for pointing out this mistake.
5) Yes, we are talking here about the scale dependent error growth on scales which are already in the nonlinear regime. Due to the initial transient after the initialization of the perturbation, we do not observe the classical exponential error growth on very small scales, but it should be there. We will explore whether by the choice of even smaller perturbations, we can reach this regime numerically.
6) Unfortunately, it is true that for the ECMWF model results, we can not clearly distinguish between our conjecture of scale dependent error growth and a previous suggestion of how to fit it and how to interpret it. We have to understand better whether this is a consequence of (in term of scale dependent error growth) too coarse scales of the EMCWF model or of too large initial errors in the error growth experiments of ECMWF.
Concerning the referee’s conclusion we agree that our study of scale dependent error growth is not exactly in the spirit of FSLE. Insprired by the referee’s comments, we calculeted the FSLE for our model, and we are considering to include the results in the revised version. However, the results of our “time after perturbation” approach do not differ qualitatively, and this is the only approach which works for the ECMWF model without modifying ECMWF’s code and runnig this model ourselves, which is out of reach to us.
-
AC1: 'Reply on RC1', Hynek Bednar, 21 Dec 2021
-
RC2: 'Comment on gmd-2021-256', Anonymous Referee #2, 10 Nov 2021
Review comments:
This well-written paper studies the error growth with a simplified toy model that extends Lorenz’ simplified system to three spatiotemporal levels. Given the critical importance of the scale-dependent nature of atmospheric predictability, this manuscript could provide some insights in our understanding of numerical weather prediction. The comparison with ECMWF data surely adds value to the study.
A major part of the study is to fit the error growth in the model with two function forms. The reviewer feels like clarifications and more analysis need to be made to justify the conclusions mentioned in the paper. Therefore, a major revision is suggested here before this paper could be published.
Specific Comments:
In the abstract, the authors wrote “there is an intrinsic limit of predictability after 22 days”, this conclusion is made by fitting the modified power law function to the ECWMF data. If the function form in Zhang et al 2019 is used, then the predictability limit becomes 15 days as mentioned in the text. The difference between these two estimated limits is sort of large. In the context of the ECMWF operational forecast system, the reviewer did not find any advantage of using the modified power law function rather than the function form in Zhang et al. Is there any reason for the reader to believe that this 22-day limit is more accurate than the 15-day limit?
Scale-dependence is the key for understanding atmospheric predictability. The authors proposed this three-scale toy model. Could it be possible to verify this three-scale model with the ECMWF data? E.g., to connect X1 with snoptical errors, X2 with meso-scale error and X3 with turbulent motions. If such filters are applied to the ECMWF data and verify the errors of different scales with the toy model, then the results would be much more convincing.
Once the parameter of the three-scale toy model is set, then its error growth behavior is also determined, Is the results shown here sensitive to the value of the parameters in the equation(e.g. F, b, c, I) ?
Line 14-15: a theoretical justification of function form in Zhang et al. is recently provided by Sun and Zhang 2020 (https://doi.org/10.1175/JAS-D-19-0271.1).
Line 180: perfect model assumption is used, right?
Line 214: what does this “initial transient behavior” look like? Is the error decreasing with time? What would happen if the initial error is further reduced towards 0?
A bracket is missing in Equation. (20)
Line 365: the extended quadratic model Eq. (21) is more accurate than the extended exponential growth? How to reconcile this with the three-scale toy model results?
Line 415-425 : The authors seem to hint that the extended power-law form is better compared to the extended quadratic form here. This is true for the toy-model. But it is not supported by the ECMWF, right?
-
AC2: 'Reply on RC2', Hynek Bednar, 22 Dec 2021
We are grateful to the referee for devoting their time to our manuscript. The valuable comments and suggestions will help us to improve the paper.
We will here respond to the main comments made:
In the abstract, the authors wrote there is an intrinsic limit of predictability
after 22 days, this conclusion is made by fitting the modified power law function to the ECWMF data. If the function form in Zhang et al 2019 is used, then the predictability limit becomes 15 days as mentioned in the text. The difference between these two estimated limits is sort of large. In the context of the ECMWF operational forecast system, the reviewer did not find any advantage of using the modified power law function rather than the function form in Zhang et al. Is there any reason for the reader to believe that this 22-day limit is more accurate than the 15-day limit?This is indeed a valid and relevant question. As we write in our manuscript (lines 416-424), from the data of error growth experiments performed with the ECMWF forecast system, it cannot be decided which law for the error growth is more appropriate. We provided some evidence that the extended power law has some theoretical justification and matches well the oberservations from our toy-model. The two competing error growth laws fitted to the ECMWF data yield different results for the prediction horizon, namely 15 versus 22 days. Due to the theoretical justifications behind the extended power law, we are inclined to follow its results and to be optimistic. Please also consider that
the current forecasts do lose their skill after about 10-15 days, and that the quadratic law is a fit to such forecasts without extrapolation to smaller initial condition errors. Our 22 days are an upper bound for the prediction horizon for hypothetical forecasts with perfect initial conditions. Also, these 22 days are in nice agreement with Krishnamurthy (2019).Scale-dependence is the key for understanding atmospheric predictability. The authors proposed this three-scale toy model. Could it be possible to verify this three-scale model with the ECMWF data? E.g., to connect X1 with synoptical errors, X2 with meso-scale error and X3 with turbulent motions. If such filters are applied to the ECMWF data and verify the errors of different scales with the toy model, then the results would be much more convincing.
We agree that it would be nice to build such a three-scale model from the ECMWF model. However, we do not think that reality has only 3 scales. The atmosphere has a large variety of spatial and temporal scales, and even a crossover from essentially 2-dimensional transport on very large spatial scales to 3-d on small scales. Also, from the practical point of view, we do not know how to do that: We could adapt and then apply our filters to the ECMWF model in order to construct 3 data sets representing different scales. But that would require a suitable fine-tuning of parameters in our filters, and therefore would be a project of its own which will be worth to be started. Our toy model only represents something like a latitude circle and not the globe. Hence, our toy-model is much simpler than any atmospheric model and serves
as a kind of didactical example of for multi-scale models. However, we have a set of similarities with real atmospheric models which we want to stress here again:
1. Our vaiable X1 has 5 to 7 main highs and lows that correspond to planetary waves (Rossby waves) and several smaller waves corresponding to synoptic-scale waves.
2. Parameters of our three-scale toy model are chosen in order for the medium scale (X2) amplitude to be approximately ten times smaller than the large scale amplitude and the small scale amplitude (X3) agan approximately ten times smaller than the medium scale amplitude. Our variables also have different oscillation periods (lines 168 170), which can be roughly compared to synoptic, meso, and turbulent scales (lines 39 - 47).
3. Bednar (2020) showed a similarity in the error growth of one-scale toy model (Eq. (2)) and ECMWF data.Once the parameter of the three-scale toy model is set, then its error growth behavior is also determined, Is the results shown here sensitive to the value of the parameters in the equation (e.g. F, b, c, I) ?
We have chosen parameters such that all levels behave chaotically (the largest Lyapunov exponent of each level is positive) and that all levels have a significant difference in amplitudes and fluctuation rates (lines 151 - 153). For such conditions, the general results are not sensitive to parameter values in quality, but the values of fit-coefficients of the different error growth laws will differ.
Line 14-15: a theoretical justification of function form in Zhang et al. is recently provided by Sun and Zhang 2020 (https://doi.org/10.1175/JAS-D-19-0271.1).
We are grateful for pointing out this reference, of which we were not aware. We changed lines 14 - 15 to: Although the quadratic hypothesis cannot be completely rejected and could serve as a first guess, the hypothesis parameters are not theoretically justifiable in the model. We added the mentioned citation: Lines 348; 351; 405 and 474-475.
Line 180: perfect model assumption is used, right?
Yes, we added this to our text (Line 182).
Line 214: what does this initial transient behavior look like? Is the error decreasing with time? What would happen if the initial error is further reduced towards 0?
Yes, before the perturbed trajectory has relaxed back to the attractor, the error is decreasing with time. When the magnitude of the initial perturbation is reduced, the relaxation time of the perturbed trajectory also decreases. We were using initial error magnitudes which were optimized for having minimal but non-zero errors after the end of the transient.
A bracket is missing in Equation. (20)
Many thanks, we corrected this.
Line 365: the extended quadratic model Eq. (21) is more accurate than the extended exponential growth? How to reconcile this with the three-scale toy model results?
Originally, the extended exponential error growth was designed to describe the error growth and error growth rate in 1-dimensional models, where “extended” means that it also captures the saturation of error growth on large scales. So the extended exponential model was made for a scale-independent error growth rate, while for our 3-scale model, we had implemented the “extension” to saturation into the power law error growth. Since the ECMWF systems exhibit multi-scale dynamics with scale dependent error growth, the extended exponential error growth model is less accurate than the extended quadratic model, but it was our intention to compare the observed error growth to the extended
power law. Bednar et al. (2020) showed that the extended quadratic model Eq. (21) is more accurate than the extended exponential growth for ECMWF data, but in that work the extended power law was not tested.Line 415-425: The authors seem to hint that the extended power-law form is better compared to the extended quadratic form here. This is true for the toymodel. But it is not supported by the ECMWF, right?
Yes, this is right, for the ECMWF model, both fits are of similar quality. In lines 415-425 we argue what we can conclude if we assume that nonetheless the extended power law is the correct description of the data.
References:
Bednar, H., Raidl, A., and Mikovsk, J.: Recalculation of error growth models parameters for the ECMWF forecast system, Geosci. Model Dev. Discuss. [preprint], https://doi.org/10.5194/gmd-2020- 250, 16 November 2020.
Krishnamurthy, V.: Predictability of Weather and Climate, AGU, 6, 7, https://doi.org/10.1029/2019EA000586, 2019.
Zhang, F., Sun, Q.: A New Theoretical Framework for Understanding Multiscale Atmospheric Predictability, J. Atmos. Sci., 77, 22972309,
https://doi.org/10.1175/JAS-D-19-0271.1, 2020.
-
AC2: 'Reply on RC2', Hynek Bednar, 22 Dec 2021
Hynek Bednář and Holger Kantz
Data sets
Prediction Error Growth in a more Realistic Atmospheric Toy Model with Three Spatiotemporal Scales Hynek Bednář http://www.doi.org/10.17605/OSF.IO/2GC9J
Model code and software
Prediction Error Growth in a more Realistic Atmospheric Toy Model with Three Spatiotemporal Scales Hynek Bednář http://www.doi.org/10.17605/OSF.IO/2GC9J
Hynek Bednář and Holger Kantz
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 402 | 69 | 17 | 488 | 3 | 4 |
- HTML: 402
- PDF: 69
- XML: 17
- Total: 488
- BibTeX: 3
- EndNote: 4
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1