Analysis of model error in forecast errors of extended atmospheric Lorenz 05 systems and the ECMWF system

Bednář, Hynek; Kantz, Holger

doi:https://doi.org/10.5194/gmd-17-6489-2024

Articles | Volume 17, issue 16

https://doi.org/10.5194/gmd-17-6489-2024

© Author(s) 2024. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/gmd-17-6489-2024

© Author(s) 2024. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 17, issue 16

Model experiment description paper

|

30 Aug 2024

Model experiment description paper |

| 30 Aug 2024

Analysis of model error in forecast errors of extended atmospheric Lorenz 05 systems and the ECMWF system

Hynek Bednář and Holger Kantz

Download

Final revised paper (published on 30 Aug 2024)
Preprint (discussion started on 04 Aug 2023)

Interactive discussion

Status: closed

RC1:
'Comment on egusphere-2023-1464', Anonymous Referee #1, 04 Sep 2023

The authors consider the question of whether omitting small spatiotemporal scales atmospheric phenomena will improve the predictability of the resulting value, using the Lorenz (2005) systems denote by L05-1, L05-2, L05-3. This system shows that omitting small spatiotemporal scales will reduce predictability more than modeling it. The authors claimed that initial errors of magnitudes that are comparable to real weather forecasts do not play a dominant role in the L05 systems, while model errors do. Moreover, they present an explanation of the observed error growth in terms of an time-averaged model error called the drift.
The authors showed that for the multi-scale systems L05-2 and L05-3, the initial error growth can be well described by the power law or the extended power law, while a simple exponential growth or the quadratic hypothesis with model error are less appropriate. We showed that in the L05 and ECMWF data the error growth in the early stage grows faster than the approximation of the whole curve, probably due to the presence of only smaller spatiotemporal scales in this part.
The paper seem to be doing a good job recalling the basic notions of initial error growth and model error growth. The L05-3 system was introduced by the authors in an earlier paper to extend the 2-scale, L05-2 system to three scales. The parameters of these systems are set so that all scales behave chaotically. Though it is not totally clear how robust the results are if the parameters are perturbed. The explanation of the initial decline and subsequent growth of the rate of model error growth by the notion of ``drift" is a nice attempt, though it is not totally clear if this is special for the L-05 systems.
Overall, I think the results are worth publishing and the efforts is a nice demonstration of illustrating the effects of different scale on error growth.
I have only a few comments for improvement of the paper:

- In the abstract, where is the claim "Generally, a

system with model error (omitting phenomena) will not improve predictability." supported in the maintext? How general is it? This seems to be a very strong statement. If not, I suggest weaken this statement.
- Although it maybe natural, it would be good to give a sentence of explanation about why choosing L05-2 and L05-3 as "reality"
- Is "this" in line 9 ``initial error growth"? Perhaps good to be more specific.
- How about adding references to the relevant figures after "as we will see in numerical simulations" in line 270?
- Would you explain why geometric mean is used rather than the usual arithmetic mean in model error growth and drift terms in (11), (14), (15), (18) ?
- The drift d(tau) at the beginning of line 271 is not defined yet, It does not seems to be the drift VECTOR in line 269. Please clarify.
- Please be consistent in terminology. For example, is ``the drift D(tau)" on Page 279 the same as d(tau) in line 271? Is it the same as the "the averaged drift D" in line 283?
- Perhaps include a table summarizing the heavy notation involved.
- The reference list will look better it it was itemized.
- I am not totally convinced (or understand) that the notion of the "drift" introduced really explain the model error growth as claimed. It seems that, taking time-average without an absolute value is similar to looking at the original system, when the system is ergodic. Perhaps the authors can explain more on what ``explain" means other than showing another summary statistics of the system.

Citation: https://doi.org/10.5194/egusphere-2023-1464-RC1
- AC1: 'Reply on RC1', Hynek Bednar, 19 Sep 2023
  
  We are grateful to the referee for devoting their time to our manuscript. The valuable comments and suggestions will help us to improve the paper.
  
  We will here respond to comments made:
  
  The parameters of these systems are set so that all scales behave chaotically. Though it is not totally clear how robust the results are if the parameters are perturbed.
  
  As long as parameters are such that all scales are chaotic, we do not expect any qualitative changes with respect to the studied scenario. We tried to ensure the robustness of the results by considering two cases of "reality" (L05-2 and L05-3 systems). Furthermore, we tested as "reality" the L05-1 system with 360 variables and as "model" the L05-1 system with 180 and 90 variables. The results are consistent with the presented results. We are aware of the need to test the results on "real" systems.
  The explanation of the initial decline and subsequent growth of the rate of model error growth by the notion of ``drift" is a nice attempt, though it is not totally clear if this is special for the L-05 systems.
  
  "Drift" was used by Orrell (2002) to explain the initial decline and subsequent growth of the rate of model error growth for the ECMWF system (500 hPa, Northern hemisphere for 10 d in October 1999 and total energy globally over a 15 d period in December 2000) . Therefore, the results do not appear to apply only to the L05 system. We have further confirmed the behavior resulting from "drift" on the ECMWF system data in Section 5.
  In the abstract, where is the claim "Generally, a system with model error (omitting phenomena) will not improve predictability." supported in the maintext? How general is it? This seems to be a very strong statement. If not, I suggest weaken this statement.
  
  We have replaced "Generally" with "In other words" (Lines 11 – 12).
  Although it maybe natural, it would be good to give a sentence of explanation about why choosing L05-2 and L05-3 as "reality"
  
  A full explanation of why L05-2 and L05-3 systems are selected as the reality is given in Section 2.2. In addition, we have added a sentence to the introduction on lines 128 – 129 ("The omitted scale is the small scale for the L05-2 system and the small and medium scale for the L05-3 system. "). Information on why we do not use the L05-2 system as model and L05-3 system as reality is given on lines 614-615.
  Is "this" in line 9 ``initial error growth? Perhaps good to be more specific.
  
  The word "product" was added to line 9.
  How about adding references to the relevant figures after "as we will see in numerical simulations" in line 270?
  
  Reference was added.
  Would you explain why geometric mean is used rather than the usual arithmetic mean in model error growth and drift terms in (11), (14), (15), (18) ?
  
  We added an explanation on line 200: "The geometric mean is chosen because of its suitability for comparison with growth governed by the largest Lyapunov exponent. For further information, see Bednar et al. (2014) or Ding and Li (2011). "
  The drift d(tau) at the beginning of line 271 is not defined yet, It does not seems to be the drift VECTOR in line 269. Please clarify.
  
  d(tau) was changed to the absolute value of drift d(tau).
  Please be consistent in terminology. For example, is ``the drift D(tau)" on Page 279 the same as d(tau) in line 271? Is it the same as the "the averaged drift D" in line 283?
  
  We improved consistency in terminology. We related D(t) to eq. (18) and d(tau) to eq. (17).
  Perhaps include a table summarizing the heavy notation involved.
  
  We itemized the numbers of the equations. We hope this will help readability.
  The reference list will look better it it was itemized.
  
  We itemized the reference list.
  I am not totally convinced (or understand) that the notion of the "drift" introduced really explain the model error growth as claimed. It seems that, taking time-average without an absolute value is similar to looking at the original system, when the system is ergodic. Perhaps the authors can explain more on what ``explain" means other than showing another summary statistics of the system.
  
  The difference is that it is the summation of vectors created from the difference in time evolution of different systems (but with the same initial conditions) after one time step. The model errors at successive time steps as vectors are not strongly correlated, and that therefore accumulating their absolute values is very different from accumulating them as vectors, where the absolute values sum will grow much faster than the vector valued sum, and that this slower error growth now gives a better explanation of the deviation of the trajectories.
  References:
  
  Bednář, H., Raidl, A., and Mikšovský, J.: Initial Error Growth and Predictability of Chaotic Low-dimensional Atmospheric Model, IJAC, 11, 256–264, https://doi.org/10.1007/s11633-014-0788-3 2014.
  
  Ding, R., Li, J.: Comparisons of two ensemble mean methods in measuring the average error growth and the predictability, Acta Meteorol Sin, 25, 395–404, https://doi.org/10.1007/s13351-011-0401-4, 2011.
  
  Orrell, D.: Role of the metric in forecast error growth: how chaotic is the weather?, Tellus, 54, 350-362, https://doi.org/10.1034/j.1600-0870.2002.01389.x, 2002.
  
  Citation: https://doi.org/10.5194/egusphere-2023-1464-AC1
RC2:
'Comment on egusphere-2023-1464', Quan Wang, 29 Oct 2023
The growth of forecast errors with respect to the lead time of atmospheric phenomena is primarily influenced by initial errors and model errors. An important question is whether omitting atmospheric phenomena, which contribute little to the final value, will improve the predictability of the resulting value. This paper uses the L05 systems and the ECMWF systems to investigate the question. Based on some numerical results, this article finds that the average prediction error grows faster in a model where small-scale phenomena are omitted, but the model error is therefore created, compared to a model where all phenomena are present, but the average forecast error growth is scale-dependent. Using these results in this paper tries to explain why omitting atmospheric phenomena, which contribute little to the final value, will not improve the predictability of the resulting value. However, this paper does not provide a complete theory to show this. Although this article says that a theory explaining and describing this behavior is developed, I did not find any strict mathematical theory in this article. In addition, there are some statement which is not clear. For instance,
Line 20: …“the instability of the system with respect to initial condition errors has grown”…, the instability is not clear?

Line 95:” …. the constant b in Eq. (5) which, irrespective of initial condition errors, will lead to a deviation of the model solution from reality…”. It seems that there is no a constant b in Eq. (5).

Line 120: “….Including small spatiotemporal scales, i.e., improving the model's spatial and temporal resolution, therefore enhances the instability with respect to initial condition error”…. the exact meaning of the instability is not clear.

Line 140: “….We measure the error magnitude e(t) after fixed time intervals ..”. there is not any expression for e(t).

5.Line 160: “…. For this scheme to be meaningful, we have to ensure that the reference trajectory is on the system's attractor and that the repetition of this scheme samples the whole attractor with correct weights (the invariant measure)….”. the existence of attractors in this system is not clear.
6.Line 195: There is no definition of .
Line 230: There is no definition of .

Line 245: There is no definition of .

For these reasons I recommend rejection.
Citation: https://doi.org/10.5194/egusphere-2023-1464-RC2
- AC2: 'Reply on RC2', Hynek Bednar, 31 Oct 2023
  
  We are grateful to the referee for devoting his time to our manuscript. We will here respond to comments made to support the validity of the article for publishing:
  
  This paper tries to explain why omitting atmospheric phenomena, which contribute little to the final value, will not improve the predictability of the resulting value. However, this paper does not provide a complete theory to show this. Although this article says that a theory explaining and describing this behavior is developed, I did not find any strict mathematical theory in this article.
  
  The developed theory is not strictly mathematical but is based on a strictly mathematical theory describing the model error growth (Drift - Section 2.4), presented by Orrell (Orrell et al., 2001; Orrell, 2002) and on a strictly mathematical theory of classical low dimensional chaos, where one observes an exponential error growth of a tiny initial error whose exponent is given by the largest Lyapunov exponent of the system. Our extension that sees Drift produced at each time step as the error of the initial conditions is based on an experiment with Lorenz L05 systems (Appendix A) and explains and describes the model error growth in this experimental setting (Section 4). The derived results are then verified in the ECMWF systems (Section 5). Because it is not a theory in a strictly mathematical sense. We replace the term “theory” with the term “hypothesis”. We believe that our hypothesis Eq. (21) is as worthy of publication as other already commonly accepted experiment-based hypotheses such as Eqs. (2)-(6).
  
  Line 20: …“the instability of the system with respect to initial condition errors has grown”…, the instability is not clear?
  
  By instability we mean the error growth rate of the initial conditions of ECMWF systems, which is expressed by the Lambda parameter from Eq. (5). The values can be seen in Figure 15a - blue curve. More details can be found in Section 5. For better understanding, we have added "(error growth rate)" to the text.
  
  Line 95:” …. the constant b in Eq. (5) which, irrespective of initial condition errors, will lead to a deviation of the model solution from reality…”. It seems that there is no a constant b in Eq. (5).
  
  b has been replaced in the text by betha. We thank the referee for spotting this misprint.
  
  Line 120: “….Including small spatiotemporal scales, i.e., improving the model's spatial and temporal resolution, therefore enhances the instability with respect to initial condition error”…. the exact meaning of the instability is not clear.
  
  By instability, we mean the error growth rate of the initial conditions. Brisch and Kantz (2019) and Zhang et al. (2019) associated initial error growth with scale-dependent error growth, where tiny errors grow much faster than larger ones. Lorenz (1969) gave a sketch of such error growth: a typical quantity to be predicted is a superposition of the dynamics on different scales. After a fast growth of the small-scale errors with saturation at these very same small scales, the large-scale errors continue to grow at a slower rate until even these saturate. We have added "(error growth rate)" to the text.
  
  Line 140: “….We measure the error magnitude e(t) after fixed time intervals ..”. there is not any expression for e(t).
  
  On line 140, e(t) is defined as the error magnitude after fixed time intervals. The expressions for the settings are shown on lines 192, 226 and 245.
  
  Line 160: “…. For this scheme to be meaningful, we have to ensure that the reference trajectory is on the system's attractor and that the repetition of this scheme samples the whole attractor with correct weights (the invariant measure)….”. the existence of attractors in this system is not clear.
  Lorenz L05 systems are widely accepted chaotic systems with a positive largest Lyapunov exponent (which is computed and presented). For L96 system (Lorenz, 1996) the existence of attractor has been shown, and because our system can be expected to be in the same model class, we expect the existence of a chaotic attractor.
  
  Line 195: There is no definition of . Line 230: There is no definition of . Line 245: There is no definition of .
  
  It is probably meant that Eqs. (11), (14) and (15) are not definitions from a strictly mathematical sence, so we replace the expression "is defined" by "is calculated".
  
  References:
  
  Brisch, J., and Kantz, H.: Power law error growth in multi-hierarchical chaotic system-a dynamical mechanism for finite prediction horizon, New J. Phys., 21, 1–7, https://doi.org/10.1088/1367-2630/ab3b4c, 2019.
  
  Lorenz, E. N.: The predictability of a flow which possesses many scales of motion, Tellus, 21, 289–307, https://doi.org/10.1111/j.2153-3490.1969.tb00444.x, 1969.
  
  Lorenz, E. N.: Predictability: a problem partly solved, in: Predictability of Weather and Climate, edited by: Palmer, T., and Hagedorn, R., Cambridge University Press, Cambridge, UK, 1–18, https://doi.org/10.1017/CBO9780511617652.004, 1996.
  
  Orrell, D., Smith, L., Barkmeijer, J., and Palmer, T. N.: Model error in weather forecasting, Nonlin. Processes Geophys., 8, 357–371, https://doi.org/10.5194/npg-8-357-2001, 2001.
  
  Orrell, D.: Role of the metric in forecast error growth: how chaotic is the weather?, Tellus, 54, 350-362, https://doi.org/10.1034/j.1600-0870.2002.01389.x, 2002.
  
  Zhang, F., Sun, Q., Magnusson, L., Buizza, R., Lin, S. H.,Chen J. H., and Emanuel K.: What is the Predictability Limit of Multilatitude Weather, J. Atmos. Sci., 76, 1077–1091, https://doi.org/10.1175/JAS-D-18-0269.1, 2019.
  
  Lorenz, E. N.: The predictability of a flow which possesses many scales of motion, Tellus, 21, 289–307, https://doi.org/10.1111/j.2153-3490.1969.tb00444.x, 1969.
  
  Citation: https://doi.org/10.5194/egusphere-2023-1464-AC2

Peer review completion

AR: Author's response | RR: Referee report | ED: Editor decision | EF: Editorial file upload

AR by Hynek Bednar on behalf of the Authors (31 Oct 2023) Author's response Author's tracked changes Manuscript

ED: Referee Nomination & Report Request started (09 Nov 2023) by Chanh Kieu

RR by Anonymous Referee #3 (04 Dec 2023)

RR by Anonymous Referee #4 (07 Dec 2023)

Suggestions for revision or reasons for rejection

Review of “Analysis of model error in forecast errors of extended atmospheric Lorenz’ 05 systems and the ECMWF system, by. H. Bednar and H. Kantz, submitted to Geoscientific Model Development.

This paper discusses the impact of omitting small-scale phenomena on error propagation in a weather forecast model. The paper concludes that “omitting small spatiotemporal scales will reduce predictability more than modeling it.” The authors then posit a hypothesis to explain the reason.

The abstract states, “This system shows that omitting small spatiotemporal scales will reduce predictability more than modeling it. In other words, a system with model error (omitting phenomena) will not improve predictability.” However, this conclusion is not new. The abstract of Jacobson (2001), for example, states, “Statistics from outer nested domains indicated that the coarser the grid spacing, the greater the underprediction of ozone.” Table 2 of the same paper quantifies the impact of grid spacing on model accuracy against data for 25 parameters, including meteorological (wind speed/direction, temperature, pressure, RH), and air quality parameters, in each of four nested domains. The paper concludes (Section 6), “For many parameters…accuracy improved from the coarsest to finest regional domains.” Please include a discussion of Jacobson (2001) in your Introduction and indicate whether any other reference you are aware of have also shown the conclusion you are making (that omitting spatiotemporal scales reduces model predictability against data) through a comparison of model results at different scales with data.

Abstract. Also, what is missing in the abstract is a summary of results relative to model resolution. How much does improving the resolution, say by a factor of 2 in each the north-south and east-west direction, reduce the error over a specified period of time?

The authors use the ECMWF model. Please clarify what parameters this model conserves. Does it conserve mass, momentum, kinetic energy, vorticity, enstrophy, and/or potential enstrophy? Do you hypothesize that the non-conservation of some of these properties may affect the results. Can you hypothesize whether results using the ECMWF would give different results from those of a different model, such as the UCLA GCM, which conserves different properties (mass, kinetic energy, vorticity, and potential enstrophy in that case)?

Along those lines, in general, do you think the conclusions drawn with this model apply to other models?

I have reviewed the responses to the other referees and find the responses reasonable.

Reference

Jacobson, M. Z., GATOR-GCMM: 2. A study of day- and nighttime ozone layers aloft, ozone in national parks, and weather during the SARMAP field campaign, J. Geophys. Res., 106, 5403-5420, 2001.

Hide

ED: Reconsider after major revisions (19 Dec 2023) by Chanh Kieu

AR by Hynek Bednar on behalf of the Authors (18 Jan 2024) Author's response Author's tracked changes Manuscript

ED: Referee Nomination & Report Request started (22 Feb 2024) by Chanh Kieu

RR by Anonymous Referee #3 (04 Mar 2024)

ED: Reconsider after major revisions (22 Mar 2024) by Chanh Kieu

AR by Hynek Bednar on behalf of the Authors (09 Apr 2024) Author's response Author's tracked changes Manuscript

ED: Referee Nomination & Report Request started (20 Apr 2024) by Chanh Kieu

RR by Anonymous Referee #3 (15 May 2024)

Suggestions for revision or reasons for rejection

The reviewer appreciates the authors' attempts to address the comments from the previous review. Nevertheless, some of the discussions remain unclear. It is advised that these issues be clarified and any confusing discussions be eliminated. A concise article with precise information would be more beneficial.

Major comments:

(A)

Concerning the physical relevance of the model, the qualitative features such as patterns, phase, and group velocities within the system fail to convince, primarily because the equation omits mechanisms like potential vorticity (PV) generation necessary for planetary waves (Rossby waves). To address this issue, it is advisable to include a disclaimer stating: "Although the equation lacks the forcing terms essential for generating planetary waves, it qualitatively mimics features similar to those of the Rossby wave."

(B)

Regarding the impact of small-scale processes, this study and the existing literature identify two types of such processes.

Type (1) involves small-scale processes introduced through an increased number of grid points or spectral modes. In this study, models with a larger number 𝑁 incorporate smaller scale processes. However, as shown in Table RR1, increasing N reduces the magnitude of the Lyapunov exponent for smaller values of 𝑁. For larger 𝑁 values (N=150, 360, 960), the Lyapunov exponents remain the same (but why?), indicating that the inclusion of smaller scale processes does not enhance instability.

In contrast, type (2) involves small scale processes through model coupling. The reviewer notes that not only do the newly introduced small scale processes but also the coupling methodology potentially impact the system's stability. [It should be noted that in real-world models, smaller spatial and temporal scale processes associated with parameterizations typically fall into this category.]

Indeed, the reviewer has previously formulated a set of generalized Lorenz models demonstrating several key insights: (1) incorporating spectral modes and additional dissipative terms at higher wavenumbers can lead to systems that only exhibit chaotic behaviors at higher critical values of the Rayleigh parameter (e.g., Shen 2019); (2) integrating smaller-scale heating processes could lead to system destabilization (e.g., Shen 2015); (3) selecting specific spectral models, akin to model coupling, can significantly influence the stability of the system.

This is why the reviewer recommended investigating (i) the impact of the coupling coefficients (ii) the potential role of nonlinear terms in generating additional critical points. Regrettably, these comment have not been addressed adequately. To remedy this, it is advised to acknowledge the following: (a) the two types of small scale processes that may generate different feedback effects, and (b) the role of coupling as another influential factor in determining the stability of the coupled system and the true impact of small scale processes on stability.

(As a result, the following statement in Abstract is not accurate:
When studying the initial error growth, it turns out that small scale phenomena, which contribute little to the forecast product, significantly affect the ability to predict this product.)

(C)

This research commenced by presuming the existence of a Lyapunov exponent (LE) and proceeded to formulate various ODEs aimed at determining the predictability horizons, emphasizing the impact of (reducing) initial conditions. However, the reviewer wishes to highlight several crucial points: (1) the LE signifies a time-averaged measure; (2) all the ODEs (with b > 0) examined in this study maintain continuous dependence on initial conditions (CDIC) across infinite time intervals, which precludes them from disclosing finite predictability for chaotic solutions. Additional insights and related discussions, which are available in Shen (2024), are provided below:

(1) For a linear ODE given by E' = sigma E, the solution E grows unboundedly. Predictability horizons can be extended by reducing initial errors or elevating thresholds.

(2) In the Logistic ODE, the solution E is bounded. However, the zero state (backward in time) and saturation value (forward in time) are only asymptotically reachable, implying that predictability horizons may also be extended by reducing initial errors or raising thresholds.

(3) For the major ODE in this study, E' = a E^{1-b} with a > 0 and Eo > 0, its solution is written as follows:

E = (Eo^b + abt)^{1/b}

Please note that the above solution is (1) unbounded for b > 0 and (2) unbounded within a finite time interval (i.e., Tmax = - Eo^b / (ab)) for b < 0 and finite Eo. It appears that only positive values of b have been considered in this study. For instance, Equation (9) becomes invalid for b < 0.

The author appreciates the discussion using the above solution that initial growth rates might not always be exponential. However, such formulations do not yield reliable predictions of predictability horizons over longer periods. More importantly, what is the relationship between the parameter "a" and the Lyapunov exponent? The same inquiry applies to the Logistic ODE: what is the connection between the parameter "sigma" and the Lyapunov exponent?

(D)

While Figure RR4 suggests the smallest growth rate for the L05-1 system and the largest for the L05-3 system, this appears inconsistent with the data from Figures 5-7, showing λex = 0.33, 0.29, and 0.46 for the L05-1, L05-2, and L05-3 systems, respectively. Could you provide any explanations for this discrepancy?

References:

Shen, B.- W., 2024: Supplementary Materials for Math340: Revisiting Lorenz’s Error Growth Models. Spring 2024. https://doi.org/10.13140/RG.2.2.36718.22089

Hide

ED: Publish subject to minor revisions (review by editor) (21 May 2024) by Chanh Kieu

AR by Hynek Bednar on behalf of the Authors (27 May 2024) Author's response Author's tracked changes Manuscript

ED: Publish as is (02 Jun 2024) by Chanh Kieu

AR by Hynek Bednar on behalf of the Authors (03 Jun 2024) Author's response Manuscript

Post-review adjustments

AA: Author's adjustment | EA: Editor approval

AA by Hynek Bednar on behalf of the Authors (21 Aug 2024) Author's adjustment Manuscript

EA: Adjustments approved (25 Aug 2024) by Chanh Kieu

Short summary

The forecast error growth of atmospheric phenomena is caused by initial and model errors. When studying the initial error growth, it may turn out that small-scale phenomena, which contribute little to the forecast product, significantly affect the ability to predict this product. With a negative result, we investigate in the extended Lorenz (2005) system whether omitting these phenomena will improve predictability. A theory explaining and describing this behavior is developed.