Received: 04 Sep 2021 – Discussion started: 06 Oct 2021
Abstract. Ensemble-variational methods form the basis of the state-of-the-art for nonlinear, scalable data assimilation, yet current designs may not be cost-effective for reducing prediction error in online, short-range forecast systems. We propose a novel, outer-loop optimization of the ensemble-variational formalism for applications in which forecast error dynamics are weakly nonlinear, such as synoptic meteorology. In order to rigorously derive our method and demonstrate its novelty, we review ensemble smoothers that appear throughout the literature in a unified Bayesian maximum-a-posteriori narrative, updating and simplifying some results. After mathematically deriving our technique, we systematically develop and inter-compare all studied schemes in the open-source Julia package DataAssimilationBenchmarks.jl, with pseudo-code provided for these methods. This high-performance numerical framework, supporting our mathematical results, produces extensive benchmarks that demonstrate the significant performance advantages of our proposed technique. In particular, our single-iteration ensemble Kalman smoother is shown both to improve prediction / posterior accuracy and to simultaneously reduce the leading order cost of iterative, sequential smoothers in a variety of relevant test cases for operational short-range forecasts. This long work is thus intended to present our novel single-iteration ensemble Kalman smoother, and to provide a theoretical and computational framework for the study of sequential, ensemble-variational Kalman filters and smoothers generally.
In this paper, the authors derive a single iteration ensemble Kalman smoother (SIEnKS) similar in spirit to the ensemble Kalman smoother (EnKS), and iterative ensemble Kalman smoother (IEnKS). They first frame popular smoothing methods as Bayesian maximum aposterioi estimators and then derive the SIEnKS through this perspective. They perform experiments on the 40 variable Lorenz-96 to demonstrate their method and compare it to EnKS, IEnKS, and a linearized IEnKS.
This is a novel and interesting idea, which the paper describes clearly. Reading this manuscript was exciting. I liked the presentation and analysis of each method. Including the pseudocodes was a great idea and helped clear questions about the implementation.
General Comments
1) It is easy to feel lost in the jargons. It would be helpful to the reader if the authors were to explain certain words more explicitly, which I have outlined in the specific comments section.
2) The experiments were done on Lorenz-96 with the 40-variable setting while observing all states. The experiments would be more compelling in an operational sense with a larger test problem and sparse observations. Are you not running these experiments purely because of the difficulty in formulating localization?
Specific Comments
1) Could you specify what is meant by outer-loop? Does this refer to the filtering step which is done first and then the inner loop of smoothing the lagged states?
2) What is meant by “online” forecast systems? Does this mean real-time?
3) In line 10 of the abstract you write “… prediction/posterior accuracy …”. I feel that the word “posterior” must be replaced by “analysis”, since prior/posterior is used with respect to the distributions while forecast/analysis is used with respect to the sample/realization.
4) In line 18, you write “four-dimensional ensemble var”. If you just mean 4D-EnVar, you should go with writing 4D-EnVar.
5) In line 40, the last word “by” should be replaced by “be” to read “... may instead BE dominated by …”.
6) In equations 28b) and 28c), you should be having (I_{N_x} + \Gamma_1^T \Gamma_1) instead of the incorrect (I_{N_x} + \Gamma_1 \Gamma_1^T) which you have. It should also be replaced in equation 36c.
Iterative optimization techniques, at the state-of-the-art of data assimilation, have largely focused on extending forecast accuracy to moderate- to long-range forecast systems. However, current methodology may not be cost-effective in reducing forecast errors in online, short-range forecast systems. We propose a novel optimization of these techniques for online, short-range forecast cycles, simultaneously providing an improvement in forecast accuracy and a reduction of the computational cost.
Iterative optimization techniques, at the state-of-the-art of data assimilation, have largely...
