Ensemble variational methods form the basis of the state of the art for nonlinear, scalable data assimilation DA;. Estimators following an ensemble Kalman filter (EnKF) analysis include the seminal maximum likelihood filter and 4DEnVAR , the ensemble randomized maximum likelihood method EnRML;, the iterative ensemble Kalman smoother IEnKS;, and the ensemble Kalman inversion EKI;. Unlike traditional 3D-Var and 4D-Var, which use the adjoint-based approximation for the gradient of the Bayesian maximum a posteriori (MAP) cost function, these EnKF-based approaches utilize an ensemble of nonlinear forecast model simulations to approximate the tangent linear model. The gradient can then be approximated by, e.g., finite differences from the ensemble mean as in the bundle variant of the IEnKS . The ensemble approximation can thus obviate constructing tangent linear and adjoint code for nonlinear forecast and observation models, which comes at a major cost in development time for operational DA systems.

These EnKF-based, ensemble variational methods combine the high accuracy of the iterative solution to the Bayesian MAP formulation of the nonlinear DA problem , the relative simplicity of model development and maintenance in ensemble-based DA , the ensemble analysis of time-dependent errors , and a variational optimization of hyperparameters for, e.g., inflation , localization , and surrogate models to augment the estimation scheme. However, while the above schemes are promising for moderately nonlinear and non-Gaussian DA, an obstacle to their use in real-time, short-range forecast systems lies in the computational barrier of simulating the nonlinear forecast model in the ensemble sampling procedure. In order to produce forecast, filter, and reanalyzed smoother statistics, these estimators may require multiple runs of the ensemble simulation over the data assimilation window (DAW), consisting of lagged past and current times.

When nonlinearity in the DA cycle is not dominated by the forecast error dynamics, as in synoptic-scale meteorology, an iterative optimization over the forecast simulation may not produce a cost-effective reduction in the forecast error. Particularly, when the linear Gaussian approximation for the forecast error dynamics is adequate, nonlinearity in the DA cycle may instead be dominated by the nonlinearity in the observation model, the nonlinearity in the hyperparameter optimization, or the nonlinearity in temporally interpolating a reanalyzed, smoothed solution over the DAW. In this setting, our formulation of iterative, ensemble variational smoothing has substantial advantages in balancing the computational cost/prediction accuracy tradeoff.

This long paper achieves three connected objectives. First, we review and update a variety of already published smoother algorithms in a narrative of Bayesian MAP estimation. Second, we use this framework to derive and contextualize our estimation technique. Third, we develop all our algorithms and test cases in the open-source Julia package DataAssimilationBenchmarks.jl . This numerical framework, supporting our mathematical results, produces extensive simulation benchmarks, validating the performance advantages of our proposed technique. These simulations likewise establish fundamental performance metrics for all estimators and the Julia package DataAssimilationBenchmarks.jl.

Our proposed technique combines the 3D sequential filter analysis and retrospective reanalysis of the classic ensemble Kalman smoother EnKS; with an iterative ensemble simulation of 4D smoothers. Following a 3D filter analysis and retrospective reanalysis of lagged states, we reinitialize each subsequent smoothing cycle with a reanalyzed, lagged ensemble state. The resulting scheme is a single-iteration ensemble Kalman smoother, denoted as such as it produces its forecast, filter, and reanalyzed smoother statistics with a single iteration of the ensemble simulation over the DAW. By doing so, we seek to minimize the leading-order cost of ensemble variational smoothing in real-time, geophysical forecast models, i.e., the ensemble simulation. However, the scheme can iteratively optimize the sequential filter cost functions in the DAW without computing additional iterations of the ensemble simulation.

We denote our framework single-iteration smoothing, while the specific implementation presented here is denoted as the single-iteration ensemble Kalman smoother (SIEnKS). For linear Gaussian systems, with the perfect model hypothesis, the SIEnKS is a consistent Bayesian estimator, albeit one that uses redundant model simulations. When the forecast error dynamics is weakly nonlinear, yet other aspects of the DA cycle are moderately to strongly nonlinear, we demonstrate that the SIEnKS has a prediction and analysis accuracy that is comparable to, and often better than, some traditional 4D iterative smoothers. However, the SIEnKS has a numerical cost that scales in iteratively optimizing the sequential filter cost functions for the DAW, i.e., the cost of the SIEnKS scales in matrix inversions in the ensemble dimension rather than in the cost of ensemble simulations, making our methodology suitable for operational short-range forecasting.

Over long DAWs, the performance of iterative smoothers can degrade significantly due to the increasing nonlinearity in temporally interpolating the posterior estimate over the window of lagged states. Furthermore, with a standard, single data assimilation (SDA) smoother, each observation is only assimilated once, meaning that new observations are only distantly connected to the initial conditions of the ensemble simulation; this can introduce many local minima to a smoother analysis, strongly affecting an optimization and references therein. To handle the increasing nonlinearity of the DA cycle in long DAWs, we derive a novel form of the method of multiple data assimilation (MDA), previously derived in a 4D stationary and sequential DAW analysis respectively. Our new MDA technique exploits the single-iteration formalism to partially assimilate each observation within the DAW with a sequential 3D filter analysis and retrospective reanalysis. Particularly, the sequential filter analysis constrains the ensemble simulation to the observations while temporally interpolating the posterior estimate over the DAW – this constraint is shown to improve the filter and forecast accuracy at the end of long DAWs and the stability of the joint posterior estimate versus the 4D approach. This key result is at the core of how the SIEnKS is able to outperform the predictive and analysis accuracy of 4D smoothing schemes while, at the same time, maintaining a lower leading-order computational cost.

This work is organized as follows. Section  introduces our notations. Section  reviews the mathematical formalism for the ensemble transform Kalman filter (ETKF) based on the LETKF formalism of , , and . Subsequently, we discuss the extension of the ETKF to fixed-lag smoothing in terms of (i) the right-transform EnKS, (ii) the IEnKS, and (iii) the SIEnKS, with each being different approximate solutions to the Bayesian MAP problem. Section discusses several applications that distinguish the performance of these estimators. Section  provides an algorithmic cost analysis for these estimators and demonstrates forecast, filter, and smoother benchmarks for the EnKS, the IEnKS, and the SIEnKS in a variety of DA configurations. Section  summarizes these results and discusses future opportunities for the single-iteration smoother framework. Appendix  contains the pseudo-code for the algorithms presented in this work, which are implemented in the open-source Julia package DataAssimilationBenchmarks.jl . Note that, due to the challenges in formulating localization/hybridization for the IEnKS , we neglect a treatment of these techniques in this initial study of the SIEnKS, though this will be treated in a future work.

The filter problem is expressed recursively in the Bayesian MAP formalism with an algorithmically stationary DAW as follows. Suppose that there is a known filter density p(x0|y0) from a previous DA cycle. Using the Markov hypothesis and the independence of observation errors, we write the filter density up to proportionality, via Bayes' law, as follows: 20ap(x1|y1:0)∝p(y1|x1,y0)p(x1,y0)20b∝p(y1|x1)︸(i)∫p(x1|x0)p(x0|y0)dx0︸(ii), which is the product of the (i) likelihood of the observation, given the forecast, and (ii) the forecast prior. The forecast prior (ii) is generated by the model propagation of the last filter density p(x0|y0), with the transition density p(x1|x0), marginalizing out x0. Given a first prior, the above recursion inductively defines the forecast and filter densities, up to proportionality, at all times.

In the perfect linear Gaussian model, the forecast prior and filter densities, 21∫p(x1|x0)p(x0|y0)dx0andp(x1|y1), are Gaussian. The Kalman filter equations recursively compute the mean x‾1fore/x‾1filt and covariance B1fore/B1filt of the random model state x1, parameterizing its distribution . In this case, the filter problem can also be written in terms of the Bayesian MAP cost function, as follows: 22J(x1)=12∥x1-x‾1fore∥B1fore2+12∥y1-H1x1∥R12. To render the above cost function into the right-transform analysis, define the matrix factor as follows: 23B1fore:=Σ1foreΣ1fore⊤, where the choice of Σ1fore can be arbitrary but is typically given in terms of a singular value decomposition SVD;. Instead of optimizing the cost function in Eq. () over the state vector x1, the optimization is equivalently written in terms of weights w, where, in the following: 24x1:=x‾1fore+Σ1forew. Thus, by rewriting Eq. () in terms of the weight vector w, we obtain the following: 25J(w)=12∥w∥2+12∥y1-H1x‾1fore-H1Σ1forew∥R12.

Furthermore, for the sake of compactness, we define the following notations: 26ay‾1:=H1x‾1fore,26bδ‾1:=R1-12y1-y‾1,26cΓ1:=R1-12H1Σ1fore. The vector δ‾1 is the innovation vector, weighted inverse proportionally to the observation uncertainty. The matrix Γ1, in one dimension with H1:=1, is equal to the standard deviation of the model forecast relative to the standard deviation of the observation error.

The cost function Eq. () is hence further reduced to the following: 27J(w)=12∥w∥2+12∥δ‾1-Γ1w∥2. This cost function is quadratic in w and can be globally minimized where ∇wJ=0. Notice that, in the following: 28∇wJ=w-Γ1⊤δ‾1-Γ1w. By setting the gradient equal to zero for w‾, we find the following expression for the optimal weights: 29a0=w‾-Γ1⊤δ‾1-Γ1w‾29b⇔Γ1⊤δ‾1=INx+Γ1⊤Γ1w‾29c⇔w‾=INx+Γ1⊤Γ1-1Γ1⊤δ‾1. From Eq. (), notice that 30∇wJ|w=0=-Γ1⊤δ‾1. Similarly, taking the gradient of Eq. (), we find that the Hessian, ΞJ:=∇w2J, is equal to the following: 31ΞJ=INx+Γ1⊤Γ1. Therefore, with w=0 corresponding to x‾1fore as the initialization of the algorithm, the MAP weights w‾ are determined by a single iteration of Newton's descent method . For iterate i, this has the general form of the following: 32wi+1:=wi-ΞJ-1∇J|w=wi.

The MAP weights define the maximum a posteriori model state as follows: 33x‾1filt:=x‾1fore+Σ1forew‾. Under the perfect linear Gaussian model assumption, J can then be rewritten in terms of the filter MAP estimate as follows: 34aJx1=12∥x1-x‾1filt∥B1filt234b⇔J(w)=12∥x‾1fore-Σ1forew-x‾1filt∥B1filt2. Define the matrix decomposition B1filt=Σ1filtΣ1filt⊤ and the change in variables as follows: 35aΩ1:=Σ1filt-1Σ1fore,35bϱ1:=Σ1filt-1x‾1fore-x‾1filt. Then, Eq. () can be rewritten as follows: 36J(w)=12∥ϱ1-Ω1w∥2. Computing the Hessian ΞJ=∇w2J from each of Eqs. () and (), we find, by their equivalence, the following: 37aINx+Γ1⊤Γ1=Ω1⊤Ω137b⇔INx+Γ1⊤Γ1=Σ1fore⊤Σ1filt-⊤Σ1filt-1Σ1fore37c⇔B1filt=Σ1foreINx+Γ1⊤Γ1-1Σ1fore⊤. If we define the covariance transform as 38T:=ΞJ-12, then this derivation above describes the square root Kalman filter recursion when written for the exact mean and covariance, which is recursively computed in the perfect linear Gaussian model. The covariance update is then as follows: 39B1filt=Σ1foreTΣ1foreT⊤. It is written entirely in terms of the matrix factor Σki and the covariance transform T, such that the background covariance need not be explicitly computed in order to produce recursive estimates. Likewise, the Kalman gain update to the mean state is reduced to Eq. () in terms of the weights and the matrix factor. This reduction is at the core of the efficiency of the ETKF in which one typically makes a reduced-rank approximation to the background covariances B1i.

Using the ensemble-based empirical estimates for the background, as in Eq. (), a modification of the above argument must be used to solve the cost function J in the ensemble span, without a direct inversion of P1fore when this is of a reduced rank. We replace the background covariance norm square with one defined by the ensemble-based covariance, as follows: 40∥v∥P1i2=Ne-1X1i†v⊤X1i†v. We then define the ensemble-based estimates as follows: 41ax1:=x^1fore+X1forew,41by^1:=H1x^1fore,41cδ^1:=R1-12y1-y^1,41dS1:=R1-12H1X1fore, where w is now a weight vector in RNe. The ensemble-based cost function is then written as follows: J̃(w)=12∥x^1fore-X1forew-x^1fore∥P1fore2+42a12∥y1-H1x^1fore-H1X1forew∥R1242b=12(Ne-1)∥w∥2+12∥δ^1-S1w∥2. Define w^ to be the minimizer of the cost function in Eq. (). demonstrate that, up to a gauge transformation, w^ yields the minimizer of the state space cost function, Eq. (), when the estimate is restricted to the ensemble span. Let Ξ̃J̃ denote the Hessian of the ensemble-based cost function in Eq. (). This equation is quadratic in w and can be solved similarly to Eq. () to render the following: 43aw^:=0-Ξ̃J̃-1∇J̃|w=0,43bT:=Ξ̃J̃-12,43cP1filt=X1foreTX1foreT⊤/(Ne-1). The ensemble transform Kalman filter (ETKF) equations are then given by the following: 44E1filt=x^1fore1⊤+X1forew^1⊤+Ne-1TU, where U∈RNe×Ne can be any mean-preserving, orthogonal transformation, i.e., U1=1. The simple choice of U:=INe is sufficient, but it has been demonstrated that choosing a random, mean-preserving orthogonal transformation at each analysis, as above, can improve the stability of the ETKF, preventing the collapse of the variances to a few modes in the empirical covariance estimate . We remark that Eq. () can be written equivalently as a single linear transformation as follows: 45aE1filt=E1foreΨ1,Ψ1:=11⊤/Ne+45bINe-11⊤/New^1⊤+Ne-1TU. The compact update notation in Eq. () is used to simplify the analysis.

If the observation operator H1 is actually nonlinear, then the ETKF typically uses the following approximation to the quadratic cost function: 46aY1:=H1E1fore,46by^1:=Y11/Ne,46cS1:=R1-12Y1-y^11⊤, where term () refers to the action of the observation operator being applied column-wise. Substituting the definitions in Eq. () for the definitions in Eq. () gives the standard nonlinear analysis in the ETKF. Note that this framework extends to a fully iterative analysis of nonlinear observation operators, as discussed in Sect. . Multiplicative covariance inflation is often used in the ETKF to handle the systematic underestimation of the forecast and filter covariance due to the sample error implied by a finite size ensemble and nonlinearity of the forecast model M1 .

The standard ETKF cycle is summarized in Algorithm . This algorithm is broken into the subroutines, in Algorithms –, which are reused throughout our analysis to emphasize the commonality and the differences in the studied smoother schemes. The filter analysis described above can be extended in several different ways when producing a smoother analysis on a DAW, including lagged past states, depending in part on whether it is formulated as a marginal or a joint smoother . The way in which this analysis is extended, utilizing a retrospective reanalysis or a 4D cost function, differentiates the EnKS from the IEnKS and highlights the ways in which the SIEnKS differs from these other schemes.

The (right-transform) fixed-lag EnKS extends the ETKF over the smoothing DAW by sequentially reanalyzing past states with future observations. This analysis is performed retrospectively in the sense that the filter cycle of the ETKF is left unchanged, while an additional smoother loop of the DA cycle performs an update on the lagged state ensembles stored in memory. Assume S=1≤L, then the EnKS estimates the joint posterior density pxL:1|yL:1 recursively, given the joint posterior estimate over the last DAW pxL-1:0|yL-1:0. We begin by considering the filter problem as in Eq. ().

Given p(xL-1:0,yL-1:0), we write the filter density up to proportionality as follows: 47ap(xL|yL:0)∝p(yL|xL,yL-1:0)p(xL,yL-1:0)∝p(yL|xL)︸(i)×47b∫p(xL|xL-1)p(xL-1:0|yL-1:0)dxL-1:0︸(ii), with the product of (i) the likelihood of the observation yL, given xL, and (ii) the forecast for xL, using the transition kernel on the last joint posterior estimate and marginalizing out xL-1:0. Recalling that p(xL|yL:1)∝p(xL|yL:0), this provides a means to sample the filter marginal of the desired joint posterior. The usual ETKF filter analysis is performed to sample the filter distribution at time tL; yet, to complete the smoothing cycle, the scheme must sample the joint posterior density p(xL:1,yL:1).

Consider that the marginal smoother density is proportional to the following: p(xL-1|yL:0)∝p(yL|xL-1,yL-1:0)×48ap(xL-1,yL-1:0)48b∝p(yL|xL-1)︸(i)p(xL-1|yL-1:0)︸(ii), where (i) is the likelihood of the observation yL, given the past state xL-1, and (ii) is the marginal density for xL-1 from the last joint posterior.

Assume now the perfect linear Gaussian model; then, the corresponding Bayesian MAP cost function is given as follows: 49J(xL-1)=12∥xL-1-x‾L-1|L-1smth∥BL-1|L-1smth2+12∥yL-HLMLxL-1∥RL2, where x‾L-1|L-1smth and BL-1|L-1smth are the mean and covariance of the marginal smoother density p(xL-1|yL-1:0). Take the following matrix decomposition: 50BL-1|L-1smth=ΣL-1|L-1smthΣL-1|L-1smth⊤. Then, write xL-1=x‾L-1|L-1smth+ΣL-1|L-1smthw, rendering the cost function as follows: J(w)=12∥w∥2+51a12∥yL-HLML(x‾L-1|L-1smth+ΣL-1|L-1smthw)∥RL251b=12∥w∥2+12∥yL-HLx‾Lfore-HLΣLforew∥RL251c=12∥w∥2+12∥δ‾L-ΓLw∥2. Let w‾ now denote the minimizer of Eq. (). It is important to recognize that 52xL:=MLx‾L-1|L-1smth+ΣL-1|L-1smthw53=x‾Lfore+ΣLforew, such that the optimal weight vector for the smoothing problem w‾ is also the optimal weight vector for the filter problem.

The ensemble-based approximation, 54axL-1=x^L-1|L-1smth+XL-1|L-1smthw,54bJ̃(w)=12(Ne-1)∥w∥2+12∥δ^L-SLw∥2, to the exact smoother cost function in Eq. () yields the retrospective analysis of the EnKS as follows: 55aw^:=0-Ξ̃J̃-1∇J̃|w=0,55bT:=Ξ̃J̃-12,EL-1|Lsmth=x^L-1|L-1smth1⊤+55cXL-1|L-1smthw^1⊤+Ne-1TU,55d≡EL-1|L-1smthΨL.

The above equations generalize for arbitrary indices k|L, completely describing the smoother loop between each filter cycle of the EnKS. After a new observation is assimilated with the ETKF analysis step, a smoother loop makes a backwards pass over the DAW, applying the transform and the weights of the ETKF filter update to each past state ensemble stored in memory. This generalizes to the case where there is a shift in the DAW with S>1, though the EnKS does not process observations asynchronously by default, i.e., the ETKF filter steps, and the subsequent retrospective reanalysis, are performed in sequence over the observations and ordered in time rather than making a global analysis over yL:L-S+1. A standard form of the EnKS is summarized in Algorithm , utilizing the subroutines in Algorithms –.

A schematic of the EnKS cycle for a lag of L=4 and a shift of S=1 is pictured in Fig. . Time moves forwards, from left to right, on the horizontal axis, with a step size of Δt. At each analysis time, the ensemble forecast from the last filter density is combined with the observation to produce the ensemble update transform ΨL. This transform is then utilized to produce the posterior estimate for all lagged state ensembles conditioned on the new observation. The information in the posterior estimate thus flows in reverse time to the lagged states stored in memory, but the information flow is unidirectional in this scheme. It is understood then that reinitializing the improved posterior estimate for the lagged states in the dynamical model does not improve the filter estimate in the perfect linear Gaussian configuration. Indeed, define the product of the ensemble transforms as follows: 56Ψk:l:=Ψk⋯Ψl. Then, for arbitrary 1≤k≤l≤L, 57aMl:kEk-1|k-1smthΨk:l=Ml:kEk-1|lsmth57b=El|k-1foreΨk:l57c=El|lsmth. This demonstrates that conditioning on the information from the observation is covariant with the dynamics. demonstrates the equivalence of the EnKS and the Rauch–Tung–Striebel (RTS) smoother, where this property of perfect linear Gaussian models is well understood. In the RTS formulation of the retrospective reanalysis, the conditional estimate reduces to the map of the current filter estimate under the reverse time model Mk-1 (; see example 7.8, chap. 7). Note, however, that both of the EnKS and ensemble RTS smoothers produce their retrospective reanalyses via a recursive ensemble transform without the need to make backwards model simulations.

The covariance of conditioning on observations and the model dynamics does not hold, however, either in the case of nonlinear dynamics or of model error. Reinitializing the DA cycle in a perfect nonlinear model with the conditional ensemble estimate E0|Lsmth can dramatically improve the accuracy of the subsequent forecast and filter statistics. Particularly, this exploits the mismatch in perfect nonlinear dynamics between ML:1E0|Lsmth≠ELfilt. Chaotic dynamics generate additional information about the initial value problem in the sense that initial conditions nearby to each other are distinguished by their subsequent evolution and divergence due to dynamical instability. Reinitializing the model forecast with the smoothed prior estimate brings new information into the forecast for states in the next DAW. This improvement in the accuracy of the ensemble statistics has been exploited to a great extent by utilizing the 4D ensemble cost function . Particularly, the filter cost function can be extended over multiple observations simultaneously and in terms of lagged states directly. This alternative approach to extending the filter analysis to the smoother analysis is discussed in the following.

The following is an up-to-date formulation of the Gauss–Newton IEnKS of and its derivations. Instead of considering the marginal smoother problem, now consider the joint posterior density directly and for a general shift S. The last posterior density is written as pxL-S:1-S|yL-S:1-S. Using the independence of observation errors and the Markov assumption recursively, p(xL:1|yL:1-S)∝∫∏k=L-S+1Lp(yk|xk)p(xk|xk-1)×58∏k=1L-Sp(xk|xk-1)px0|yL-S:1-Sdx0. Additionally, using the perfect model assumption, 59p(xk|xk-1)=δxk-Mkxk-1 for every k. Therefore, p(xL:1|yL:1-S)∝∫p(x0|yL-S:1-S)︸(i)×∏k=L-S+1Lp(yk|xk)︸(ii)×60∏k=1Lδxk-Mkxk-1︸(iii)dx0, where term (i) in Eq. () represents the marginal smoother density for x0|L-S over the last DAW, term (ii) represents the joint likelihood of the observations given the model state, and term (iii) represents the free forecast of the smoother estimate for x0|L-S. Noting that p(xL:1|yL:1)∝p(xL:1|yL:1-S), this provides a recursive form to sample the joint posterior density.

Under the perfect linear Gaussian model assumption, the above derivation leads to the following exact 4D cost function: 61Jx0:=12∥x0-x‾0|L-Ssmth∥B0|L-Ssmth2+12∑k=L-S+1L∥yk-HkMk:1x0∥Rk2. The ensemble-based approximation, using notations as in Eq. (), yields the following: 62ax0:=x^0|L-Ssmth+X0|L-Ssmthw,J̃(w):=12(Ne-1)∥w∥2+62b12∑k=L-S+1L∥δ^k-Skw∥2. Notice that Eq. () is quadratic in w; therefore, for the perfect linear Gaussian model, one can perform a global analysis over all new observations in the DAW at once.

The gradient and the Hessian of the ensemble-based 4D cost function are given as follows: 63a∇J̃:=(Ne-1)w-∑k=L-S+1LSk⊤δ^k-Skw,63bΞ̃J̃:=(Ne-1)INe+∑k=L-S+1LSk⊤Sk, so that, evaluating at w=0, the minimizer w^ is again given by a single iteration of Newton's descent 64w^:=0-Ξ̃J̃∇J̃|w=0. Define the covariance transform again as T:=Ξ̃J̃-12. We denote the right ensemble transform corresponding to the 4D analysis ΨL-S+1:L4D to distinguish from the product of the sequential filter transforms ΨL-S+1:L. The global analyses are defined as follows: ΨL-S+1:L4D:=11⊤/Ne+65aINe-11⊤/New^1⊤+Ne-1TU,65bE0|Lsmth=E0|L-SsmthΨL-S+1:L4D, where U is any mean-preserving orthogonal matrix.

In the perfect linear Gaussian model, this formulation of the IEnKS is actually equivalent to the 4D-EnKF of , , and . The above scheme produces a global analysis of all observations within the DAW, even asynchronously from the standard filter cycle . One generates a free ensemble forecast with the initial conditions drawn iid as p(x0|yL-S:1-S), and all data available within the DAW are used to estimate the update to the initial ensemble. The perfect model assumption means that the updated initial ensemble E0|Lsmth can then be used to temporally interpolate the joint posterior estimate over the entire DAW from the marginal sample, i.e., for any 0<k≤L, a smoothing solution is defined as follows: 66Mk:1E0|L-SsmthΨL-S+1:L4D≡Ek|Lsmth.

When Mk and Hk are nonlinear, the IEnKS formulation is extended with additional iterations of Newton's descent, as in Eq. (), in order to iteratively optimize the update weights. Specifically, the gradient is given by the following: 67a∇J̃:=Ne-1w-∑k=L-S+1LỸk⊤Rk-1φ,67bφ:=yk-Hk∘Mk:1x^0|L-Ssmth+X0|L-Ssmthw, where Ỹk represents a directional derivative of the observation and state models with respect to the ensemble perturbations at the ensemble mean, as follows: 68Ỹk:=∇|x^0|L-SsmthHk∘Mk:1X0|L-Ssmth. This describes the sensitivities of the cost function, with respect to the ensemble perturbations, mapped to the observation space. When the dynamics is weakly nonlinear, the ensemble perturbations of the EnKS and IEnKS are known to closely align with the span of the backward Lyapunov vectors of the nonlinear model along the true state trajectory . Under these conditions, Eq. () can be interpreted as a directional derivative with respect to the forecast error growth along the dynamical instabilities of the nonlinear model (see , and references therein).

In order to avoid an explicit computation of the tangent linear model and the adjoint as in 4D-Var, and proposed two formulations to approximate the tangent linear propagation of the ensemble perturbations. The bundle scheme makes an explicit approximation of finite differences in the observation space where, for an arbitrary ensemble, they define the approximate linearization as follows: 69Yk:=1ϵHk∘Mk:1x01⊤+ϵX0INe-11⊤/Ne, for a small constant ϵ. Alternatively, the transform version provides a different approximation to the variational analysis, using the covariance transform T and its inverse as a pre-/post-conditioning of the perturbations used in the sensitivities approximation. The transform variant of the IEnKS is in some cases more numerically efficient than the bundle version, requiring fewer ensemble simulations, and it is explicitly related to the ETKF/EnKS/4D-EnKF formalism presented thus far. For these reasons, the transform approximation is used as a basis of comparison with the other schemes in this work.

For the IEnKS transform variant, the ensemble-based approximations are redefined in each Newton iteration as follows: 70aYk:=HkEk,70by^k:=Yk1/Ne,70cSk:=Rk-12Yk-y^k1⊤T-1,70dδ^:=Rk-12yk-y^k, where the first covariance transform is defined as T:=INe, the gradient and Hessian are computed as in Eq. () from the above, and where the covariance transform is redefined in terms of the Hessian, T:=Ξ̃J̃-12, at the end of each iteration. With these definitions, the first iteration of the IEnKS transform variant corresponds to the solution of the nonlinear 4D-EnKF, but subsequent iterates are initialized by pre-conditioning the initial ensemble perturbations via the update T and post-conditioning the sensitivities by the inverse transform T-1.

An updated form of the Gauss–Newton IEnKS transform variant is presented in Algorithm . Note that, while Algorithm  does not explicitly reference the sub-routine in Algorithm , many of the same steps are used in the IEnKS when computing the sensitivities. It is important to notice that, for S>1, the IEnKS only requires a single computation of the square root inverse of the Hessian of the 4D cost function, per iteration of the optimization, to process all observations in the DAW. On the other hand, the EnKS processes these observations sequentially, requiring S total square root inverse calculations of the Hessian, corresponding to each of the sequential filter cost functions.

The IEnKS is computationally constrained by the fact that each iteration of the descent requires L total ensemble simulations in the dynamical state model Mk. One can minimize this expense by using a single iteration of the IEnKS equations, which is denoted the linearized IEnKS (Lin-IEnKS) by . When the overall DA cycle is nonlinear, but only weakly nonlinear, this single iteration of the IEnKS algorithm can produce a dramatic improvement in the forecast accuracy versus the forecast/filter cycle of the EnKS. However, the overall nonlinearity of the DA cycle may be strongly influenced by factors other than the model forecast Mk itself. As a simple example, consider the case in which Hk is nonlinear yet Mk≡Mk for all k. In this setting, it may be more numerically efficient to iterate upon the 3D filter cost function rather than the full 4D cost function which requires simulations of the state model. Combining (i) the filter step and retrospective reanalysis of the EnKS and (ii) the single iteration of the ensemble simulation over the DAW as in Lin-IEnKS, we obtain an estimation scheme that sequentially solves the nonlinear filter cost functions in the current DAW, while making an improved forecast in the next by transmitting the retrospective analyses through the dynamics via the updated initial ensemble.

Just as the IEnKS extends the linear 4D cost function, the filter cost function in Eq. () can be extended with Newton iterates in the presence of a nonlinear observation operator. The maximum likelihood ensemble filter (MLEF) of and is an estimator designed to process nonlinear observation operators and can be derived in the common ETKF formalism. Particularly, the algorithm can be granted bundle and transform variants like the IEnKS (; see Sect. 6.7.2.1), which are designed to approximate the directional derivative of the nonlinear observation operator with respect to the forecast ensemble perturbations at the forecast mean, 73Ỹk:=∇|x^kforeHkXkfore, which is used in the nonlinear filter cost function gradient as follows: ∇J̃:=Ne-1w-74Ỹk⊤Rk-1yk-Hkx^kfore+Xkforew.

When the forecast error dynamics is weakly nonlinear, the MLEF-style nonlinear filter cost function optimization provides a direct extension to the SIEnKS. The transform, as defined in the sub-routine in Algorithm , is interchangeable with the usual ensemble transform in Algorithm . In this way, the EnKS and the SIEnKS can each process nonlinear observation operators with an iterative optimization in the filter cost function alone and, subsequently, apply their retrospective analyses as usual. We refer to the EnKS analysis with MLEF transform as the maximum likelihood ensemble smoother (MLES), though we refer to the SIEnKS as usual, whether it uses a single iteration or multiple iterations of the solution to the filter cost function. Note that only the transform step needs to be interchanged in Algorithms  and , so we do not provide additional pseudo-code.

Consider that, for the MLES and the SIEnKS, the number of Hessian square root inverse calculations expands in the number of iterations used in Algorithm  to compute the transform for each of the S observations in the DAW. For each iteration of the IEnKS, this again requires only a single square root inverse calculation of the 4D cost function Hessian. However, even if the forecast error dynamics is weakly nonlinear, optimizing versus the nonlinear observation operator requires L ensemble simulations for each iteration used to optimize the cost function.

Due to the bias of Kalman-like estimators in nonlinear dynamics, covariance inflation, as in Algorithm , is widely used to regularize these schemes. In particular, this can ameliorate the systematic underestimation of the prediction/posterior uncertainty due to sample error and bias. Empirically tuning the multiplicative inflation coefficient λ≥1 can be effective in stationary dynamics. However, empirically tuning this parameter can be costly, potentially requiring many model simulations, and the tuned value may not be optimal across timescales in which the dynamical system becomes non-stationary. A variety of techniques is used in practice for adaptive covariance estimation, inflation, or augmentation, accounting for these deficiencies of the Kalman-like estimators (, and references therein).

One alternative to empirically tuning λ is to derive an adaptive multiplicative covariance inflation factor via a hierarchical Bayesian model by including a prior on the background mean and covariance px‾1fore,B1fore, as in the finite size formalism of , , and . This formalism seeks to marginalize out over the first 2 moments of the background, yielding a Gaussian mixture model for the forecast prior as follows: px1|E1fore=∫px1|E1fore,x‾1fore,B1fore×75px‾1fore,B1fore|E1foredx‾1foredB1fore. Using Jeffreys' hyperprior for x‾1fore and B1fore, the ensemble-based filter MAP cost function can be derived as proportional to the following: J̃(w):=12∥y-Hx^1fore+X1forew∥R12+76Ne2log⁡ϵNe+∥w∥2, where ϵNe:=1+1Ne. Notice that Eq. () is non-quadratic in w, regardless of whether H1 is linear or nonlinear, such that one can iteratively optimize the solution to the nonlinear filter cost function with a Gauss–Newton approximation of the descent. When accounting for the nonlinearity in the ensemble evolution and the sample error due to small ensemble sizes in perfect models, optimizing the extended cost function in Eq. () can be an effective means to regularize the EnKF. In the presence of significant model error, one may need to extend the finite size formalism to the variant developed by .

Algorithm  presents an updated version of the finite size ensemble Kalman filter (EnKF-N) transform calculation of , explicitly based on the IEnKS transform approximation of the gradient of the observation operator. The hyperprior for the background mean and covariance is similarly introduced to the IEnKS and optimized over an extended 4D cost function. Note that, in the case when Hk≡Hk is linear, a dual, scalar optimization can be performed for the filter cost function with less numerical expense. However, there is no similar reduction to the extended 4D cost function, and in order to emphasize the structural difference between the 4D approach and the sequential approach, we focus on the transform variant analogous to the IEnKS optimization.

Extending the adaptive covariance inflation in the finite size formalism to either the EnKS or the SIEnKS is simple, requiring that the ensemble transform calculation is interchanged with Algorithm  and that the tuned multiplicative inflation step is eliminated. The finite size iterative ensemble Kalman smoother (IEnKS-N) transform variant, including adaptive inflation as above, is described in Algorithm . Notice that iteratively optimizing the inflation hyperparameter comes at the additional expense of square root inverse Hessian calculations for the EnKS and the SIEnKS, while the IEnKS also requires L additional ensemble simulations for each iteration.

When the lag L>1 is long, temporally interpolating the posterior estimate in the DAW via the nonlinear model solution, as in Eq. (), becomes increasingly nonlinear. In chaotic dynamics, the small simulation errors introduced this way eventually degrade the posterior estimate, and this interpolation becomes unstable when L is taken to be sufficiently large. Furthermore, for the 4D cost function, observations only distantly connected with the initial condition at the beginning of the DAW render the cost function with more local minima that may strongly affect the performance of the optimization. Multiple data assimilation is a commonly used technique, based on statistical tempering , designed to relax the nonlinearity of performing the MAP estimate by artificially inflating the variances of the observation errors with weights and assimilating these observations multiple times. Multiple data assimilation is made consistent with the Bayesian posterior in perfect linear Gaussian models by appropriately choosing weights so that, over all times that an observation vector is assimilated, all of its associated weights sum to one . Given Gaussian likelihood functions, this implies that the sum of the precision matrices over the multiple assimilation steps equals R-1, as with the usual Kalman filter update.

Multiple data assimilation is integrated into the EnRML for static DAWs in reservoir modeling and references therein. With the fixed-lag, sequential EnKS, there is no reason to perform MDA as the assimilation occurs in a single pass over each observation with the filter step as in the ETKF. Sequential MDA, with DAWs shifting in time, was first derived with the IEnKS by . In order to sample the appropriate density, the IEnKS MDA estimation is broken over two stages. First, in the balancing stage, the IEnKS fully assimilates all partially assimilated observations, targeting the joint posterior statistics. Second, the window of the partially assimilated observations is shifted in time with the MDA stage. The SIEnKS is similarly broken over these two stages, using the same weights as the IEnKS above. However, there is an important difference in the way MDA is formulated for the SIEnKS versus the IEnKS. For the SIEnKS, each observation in the DAW is assimilated with the sequential 3D filter cost function instead of the global 4D analysis in the IEnKS. The sequential filter analysis constrains the posterior's interpolation estimate to the observations in the balancing stage, as observations are assimilated sequentially in the SIEnKS, whereas the posterior estimate is performed by interpolating with a free forecast from the marginal posterior estimate in the IEnKS. Our novel SIEnKS MDA scheme is derived as follows.

Recall our algorithmically stationary DAW, {t1,⋯,tL}, and suppose, at the moment, that there is a shift of S=1 and an arbitrary lag L. We take the notation that the covariance matrices for the likelihood functions are inflated to be as follows: 77pyβ|x:=ny|Hx,β-1R, where the observation weights are assumed 0<β≤1. We index the weight for observation yk at the present time tL as βk|L. For consistency with the perfect linear Gaussian model, we require that 78∑i=1Lβi|L=1. This implies that, as we assimilate an observation vector for L total times, shifting the algorithmically stationary DAW, the sum of the weights used to assimilate the observation equals one.

We denote 79αk|L:=∑i=kLβi|L as the fraction of the observation yk that has been assimilated after the analysis step at the time tL. Note that, under the Gaussian likelihood assumption, and assuming the independence of the fractional observations, this implies that 80∏i=kLpyβi|L|x=pyαk|L|x.

Let βl:k|L and αl:k|L denote the length (l-k+1) vectors as follows: 81aβl:k|L=βl|L⋯βk|L,81bαl:k|L=αl|L⋯αk|L. We then define the sequences, 82ayl:kβl:k|L:=ylβl|L,yl-1βl-1|L,⋯,ykβk|L,82byl:kαl:k|L:=ylαl:L,yl-1αl-1|L,⋯,ykαk|l, as the observations yl:k in the current DAW {t1,⋯,tL}, with Eq. (), the corresponding MDA weights for this DAW, and, with Eq. (), the total portion of each observation assimilated in the MDA conditional density for this DAW after the analysis step. Similar definitions apply with the indices l:k|L-1 but are relative to the previous DAW.

For the current DAW, the balancing stage is designed to sample the joint posterior density, 83pxL:1|yL:1, where the current cycle is initialized with a sample of the MDA conditional density, 84px0|yL-1:0αL-1:0|L-1. That is, from the previous cycle, we have a marginal estimate for x0, given the sequence of observations yL-1:0, where the portion of observation yk that has been assimilated already is given by αk|L-1. Notice that α0|L-1=1 so that y0 has already been fully assimilated. To fully assimilate y1, we note that 1-α1|L-1=β1|L, and therefore, px1:0|yL-1:2αL-1:2|L-1,y1:0∝py1β1|L|x1px1|x0×85px0|yL-1:0αL-1:0|L-1. The above corresponds to a single simulation/analysis step in an EnKS cycle, where the observation y1β1|L is assimilated, and a retrospective reanalysis is applied to the ensemble at t0.