Data assimilation (DA) methodologies provide a conceptual framework to integrate the information content embedded in observations and numerical models, and play a pivotal role in deriving accurate estimates of the state of Earth systems components and reducing the uncertainties of geophysical systems among others, see the methods and applications reviewed in.

Thanks to the scalability of parallel implementations, the use of ensemble algorithms see e.g.,and references therein has been proposed to estimate uncertainty and improve assimilation skills in Kalman filters and variational methodologies. On the other hand, some of the strong points of ensemble and variational methods have been merged in hybrid filters e.g.,. Moreover, recent developments of EnKF have been conceived to face increasing nonlinearities and model complexity that are also related to the expanding availability of observations and computational resources as discussed in, e.g.,.

However, the definition of the strategy for ensemble generation is not a trivial task see e.g.,. Straightforward Monte Carlo approaches are not usually a viable option in geoscience applications, because they would require too large a number of ensemble members, and consequently, computational effort. The number of ensemble members can be reduced by adopting deterministic sampling methods as opposed to stochastic EnKF methods, see e.g.and the references therein.

Common examples of deterministic EnKF are SEIK and ETKF . According to , these are second-order exact methods, where the term order refers to polynomial exactness (in contrast with the order convergence in ensemble size in Monte Carlo stochastic methods or the time-integration order in numerical methods). We want to extend this concept by introducing here the notion of polynomial order of approximation of the filter or of its sampling method or, more concisely, order: a hth-order ensemble has the property of providing a forecast mean with no error if a hth-order polynomial model is used for forecasting. According to this definition, SEIK and ETKF are second-order methods, while Monte Carlo sampling has order zero. As proven in Appendix , hth-order is achieved by sampling an ensemble that matches the first h statistical moments of the uncertainty probability density function (pdf) before evolution (also called prior in Bayesian statistics).

Most of the models used in geoscience applications are based on systems of differential equations that cannot be represented by a second-order polynomial and in all of these cases the second-order sampling methods provide a non-exact estimation of the forecast mean, which is affected by an error strictly related to the approximation error that would be made if approximating the model by a second-order polynomial. Furthermore, the second-order approximation of the forecast mean is more effective the closer the ensemble members are to each other (i.e., small uncertainty), thus the higher the uncertainty the worse will be the approximation error in the mean computation. Since the state estimation is often affected by a relatively high uncertainty in geosciences data assimilation applications, this approximation error may not be negligible.

A potential strategy to reduce this error is the use of a higher order of approximation. In general, this would require a larger ensemble with respect to the second-order methods and consequently larger computational costs. Indeed, a higher order of approximation implies a larger number of ensemble members to represent the same uncertainty subspace

In the space of the possible state vectors, the uncertainty subspace is the subspace generated by the ensemble members (i.e., the smallest subspace that contains all the ensemble members). The uncertainty subspace was originally called error subspace seefor an introduction to the error subspace concept, but in the context of this work it is more natural to interpret the ensemble as a proxy of the uncertainty, avoiding unnecessary confusion with, for instance, the approximation error.

In this work we propose a novel method to generate ensembles for data assimilation applications. The preliminary idea is that the more statistical moments are shared by an uncertainty pdf and an ensemble representing it, the lower the error made by the ensemble in the forecast mean (Appendix ). In this sense, an ensemble of order h, that is characterized by the property of matching the uncertainty pdf up to the moment of order h, provides a more accurate forecast mean than an ensemble of order lower than h. Compared to deterministic second-order square-root sampling methods, whose ensembles match the first 2 statistical moments of the uncertainty pdf, the novel strategy achieves an ensemble with a higher order of approximation (i.e., matching more than 2 statistical moments) without increasing the ensemble size. The strategy is based on the choice, among all the second-order ensembles, of those that feature a higher order of approximation along the principal components of the uncertainty (i.e., those directions where the ensemble spread is larger and the approximation is consequently worse).

Figure shows an uncertainty represented by a 3-dimensional standard normal distribution in an xyz Cartesian coordinate system (top of Fig. where spherical uncertainty isosurfaces are represented by shadowed areas). According to the second-order-exact sampling , four opportunely chosen ensemble members are needed to represent this uncertainty probability density function up to the second order. The ensemble members are not uniquely determined by the second-order exact sampling, which indeed allows for random rotations. In the xyz Cartesian coordinate system of Fig. , four ensemble members that ensure the second-order exact sampling (yellow points at the vertices of a regular tetrahedron) are chosen among all the possible second-order sampling (each corresponding to a rotation of the tetrahedron). Thanks to their particular orientation, the four vertices draw (project) a square (a non-skewed shape) in the xy plane (middle panel of Fig. ; Appendix ), achieving a third-order approximation in the xy plane. Indeed, four square-shaped ensemble members provide a third-order approximation for a 2-dimensional normal distribution (Appendix ). Similarly, there are other possible choices of second-order vertices that project a square (third-order approximation) in the xy plane, but thanks to this particular orientation, the four members chosen in Fig. can be further projected along the x-axis to obtain the special 3-point distribution (shown at the bottom of Fig. ), where the central projected point weights as two of the original points. As it is, this 1-dimensional distribution does not achieve an order higher than 3 (already achieved in the xy plane with the square projection) along the x axis but three properly weighted ensemble members can reach an approximation order as high as 5 on the x axis (not shown in Fig. , see Appendix ). On the contrary, if proper weights are not assigned, the highest approximation order that can be obtained using 4 members in the 1-dimensional case is limited to 3. In summary, Fig. represents an example of an ensemble with approximation order 2 in the xyz 3-dimensional space, which is opportunely oriented to have approximation order 3 in the xy plane and, by assigning proper weights, can reach approximation order 5 along the x axis (Appendix ). Generally, the possibility to increase the approximation order using appropriate rotations and weights opens the way to build an ensemble with a higher order of approximation along dimensions where uncertainties are higher. For instance, in the case of a non-standard normal distribution (in contrast with the 3-dimensional standard distribution of Fig. ), if the orientation of the xyz coordinate system was chosen so that the z component of the distribution was less relevant than the x and y components, then the ensemble built with the third-order approximation would mitigate the approximation error in the x and y dimensions where the spread is higher. Similarly, if the coordinate system is oriented so that the variance is larger on x than y, then the achieved fifth-order approximation reduces the error more efficiently where the spread is maximum (x coordinate). As shown later in this work, the high-order sampling mimics this strategy in the ensemble data assimilation context, in order to achieve a higher order in the uncertainty principal components.

The high-order sampling generalizes the idea described above considering that any probability distribution (under reasonable hypothesis) can be sampled with high order of approximation with a weighted ensemble that, in the special case of the Gaussian distribution, is represented by the nodes and weights of the Gauss-Hermite quadrature rule chap. 2. Once the approximation order (h larger than 2) and the ensemble size (r+1) are fixed, the hth-order weighted ensemble is computed for a relatively small s-dimensional subspace (s lower than r); then, a new (r+1)-sized second-order weighted ensemble is computed such that its projections along the uncertainty principal components coincide with the hth-order ensemble. The resulting ensemble has a second-order approximation in the r-dimensional subspace and a hth-order approximation in the s-dimensional subspace of the principal uncertainty directions. By comparison, a second-order-exact sampling (e.g., SEIK) with r+1 ensemble members would provide a second-order approximation for the whole r-dimensional subspace.

It is worth noting that this approach is beneficial independently of the uncertainty pdf. Even in the Gaussian case, where the uncertainty pdf is uniquely determined by its mean and covariance, the ensemble is not uniquely determined in the same way. Thus, building an ensemble matching also higher Gaussian moments helps to better represent the uncertainty pdf, which in turn translates in a smaller error on the forecast mean.

The two phases of the high-order sampling (initialization and sampling) are described in the following from an algorithmic point of view. The initialization includes the steps that must be executed only once, while the sampling presents the ensemble generation procedure. In the sampling description, the system of equations to compute the ensemble perturbations and weights is presented. As the ensemble moments are used in the sampling phase but are calculated only once, their calculation is presented in the initialization section. Mathematical explanations can be found in Appendix .

Similar to other data assimilation ensemble schemes, the GHOSH filter provides an estimate of the state of a system at some discrete times ti in terms of the state vector and the covariance matrix representing the estimation uncertainty. Hence, at time ti, a forecast is composed of the forecast state vector xif∈RN (N is the dimension of the state space) and the forecast uncertainty covariance matrix Pif. If an observation vector yi is available at time ti, the information is assimilated in the analysis state vector xia with uncertainty covariance matrix Pia.

Being an ensemble-based filter scheme, the GHOSH filter represents these quantities by an ensemble of state vectors, e.g., 9Xif=xif,1,…,xif,r+1 of r+1 model state realizations. However, unlike other ensemble filters that uniformly weight the ensemble members, the GHOSH filter assigns to Xif a vector of corresponding weights 10w=w1⋮wr+1. The state estimate is given by the weighted mean 11xif=∑j=1r+1xif,jwj=Xifw, while the uncertainty covariance is approximated by the ensemble covariance matrix 12Pif≈XifWXifT-xifxifT, with W=diagw being the diagonal matrix of weights. As in other ensemble-based filter schemes, the uncertainty covariance never needs to be explicitly computed. However, Eq. () and the like (e.g., Eqs. , , , ) are still shown in this form for convenience.

The GHOSH algorithm consists of 5 phases, i.e., initialization, forecast, forecast high-order sampling, analysis and analysis high-order sampling (Fig. ). The initialization is performed once at the beginning of the filter application, the forecast and forecast high-order sampling phases are carried out at each time ti, and analysis and analysis high-order sampling are executed only when observations are available.

The analysis equations are a novel weighted version of the ensemble Kalman filter equations , while the forecast phase equations rely on a hybrid approach. In fact, the forecast uncertainty is obtained by combining the ensemble covariance (weighted by a forgetting factor) and a parametric covariance matrix Qi, which is built from the existing knowledge of the system, as done, for instance, for the background matrix in many variational schemes .

The method includes two resampling phases, after analysis and after forecast (Fig. ), at which the whole ensemble is rebuilt using the high-order sampling method (Sect. ). The resampling produces a sample of state vectors that matches the moments of the uncertainty probability density function with better precision than in the other commonly used ensemble DA algorithms, resulting in an order of approximation greater than 2 (see Appendix ) in the principal uncertainty modes. In this way the GHOSH filter takes advantage of the high approximation order of the sampling method twice: before applying the model operator and before applying the observation operator.

Table helps taking track of symbols and notation.

The GHOSH filter has been tested in a very large set of twin experiments based on the Lorenz96 model to evaluate the performance of GHOSH compared with a second-order filter (SEIK). The Lorenz96 model, which has a chaotic behaviour that is comparable to that of fluid dynamics equations at a much lower computational cost, is a standard choice for testing new data assimilation schemes e.g.,.

In each experiment, the truth trajectory is provided by integrating the model for a certain time interval. Then, at regular time intervals, observations of a subset of the system variables have been extracted from the truth, adding a random error to each of them to represent the observation uncertainty. The same set of observations is then used for two assimilation experiments, one using the SEIK filter as a second-order approximation filter, and one using the GHOSH filter with fifth-order approximation (i.e., h=5, see Sect. ). The SEIK and GHOSH assimilation experiments are initialized with the same initial condition, chosen randomly around the truth initial condition. After an initial spin-up of 10 time units, the skill of each filter is evaluated by computing, during the whole simulation, the root mean square error (RMSE) between the filter forecast and the truth before the assimilation step. Representing both how often and how far the filters deviate from the truth, RMSE is a common proxy for filter skill. In the experiments, the RMSEs on assimilated and non-assimilated variables have been evaluated for each filter. Separately assessing filter skills on non-assimilated variables provides indications on the capability of transferring information gathered with observations to the whole state of the system, exploiting correlations.

In order to produce reliable statistics, the same procedure has been repeated 90 000 times varying the truth trajectory, the observations, the filters' initial conditions and some filters hyper-parameters such as the ensemble size and the forgetting factor.

The name “Gauss-Hermite high-order sampling hybrid filter” was chosen to indicate the two main features of this novel filter. First, it is a hybrid filter, in the sense that the uncertainty covariance matrix is obtained by averaging the ensemble covariance and a parametric covariance matrix non-depending on the ensemble, as described in, e.g., . Second, it exploits a novel sampling method based on a Gauss-Hermite-like quadrature rule to reach an arbitrarily high (depending on the ensemble size) polynomial order of approximation.

The latter feature introduces, to the best of our knowledge, a completely new class of DA algorithms with an order of approximation higher than 2. The advantage of moving to a higher order of approximation has previously been documented in the literature, in particular compares order 1 algorithms e.g., SEEK and order 2 algorithms e.g., SEIK. The results of Sect. confirm the expected benefits of the high-order sampling adopted in the GHOSH filter. In particular, in the Lorenz96 idealized and controlled conditions, the novel method outperforms (up to 56 % lower RMSE) a typical second-order method like SEIK and features higher stability and better performance.

Thanks to the large number of settings tested in the Lorenz96 twin experiment, it is possible to suggest the conditions where the GHOSH filter considerably increases the assimilation skill and reliability. Concerning the ensemble size, the larger improvements occur when the dimension of the uncertainty subspace spanned by the ensemble is large enough to be representative. If the number of ensemble members is too small, the assimilation simply fails independently of the tested assimilation algorithm, but even in that case, the GHOSH filter error is less detrimental than the SEIK error. Moreover, it has been shown that the GHOSH increased accuracy reduces instabilities and numerical divergence in our Lorenz96 implementation (Fig. ). This aspect is consistent with the fact that a higher order of approximation helps to manage nonlinearities . GHOSH filter showed a lower need for inflation with respect to SEIK, removing one of the instability causes observed in the twin experiment (Sect. ). Indeed, in the twin experiment, a larger number of instability cases were observed at lower inflation values both for SEIK and GHOSH (one example with diverging SEIK is shown in Fig. ). The lower need for inflation can be also seen as a GHOSH's improved capacity to take into account nonlinearity. Indeed, nonlinearity typically introduces errors that are not considered by ensemble filters, which need inflation to compensate for this uncertainty covariance underestimation see e.g.,. Finally, the GHOSH filter could remarkably perform a successful assimilation even when the SEIK filter is failing due to observation sparsity.

Even if a large number of settings have been tested in the twin experiments, other Lorenz96 literature applications see e.g., tested ensemble DA methods applying different conditions and strategies. In particular, longer time series, different observation operators or different state vector dimensions have been considered. The shorter time series in the first experiment phase in the present work (Sect. ) are motivated by the aim to focus on comparing DA convergence in the phase after the assimilation and not on long-term convergence. Thanks to this approach, our results provide information for possible realistic applications of GHOSH in geoscientific operational simulations. In this framework, the shown initial fast convergence of the GHOSH filter is strongly beneficial to improve the simulation accuracy on temporal scales of interest.

In addition to a higher polynomial order, the GHOSH filter features a resampling taking place twice per step. Ensemble Kalman-like filters usually adopt an interpolation strategy by applying the observation operator directly to the X̃if ensemble of the states after the model evolution (see Eq. ). However, this strategy might lead to inaccurate estimations because the covariance matrix Qi of Eq. () is not taken into account in the interpolation. For this reason, if Qi is not zero, GHOSH applies the observation operator to a new ensemble, resampled between the forecast and analysis phases (Fig. ). In this way, Qi is taken into account and, especially in the case of a nonlinear observation operator, the high order of the sampling method is exploited to improve the accuracy of the projection into the observation space as in Part 2, .

The statistical moments μ in Eq. () are key hyper-parameters that describe the uncertainty pdf moments for orders higher than 2. In our experiments we used the moments of a Gaussian distribution, but it is worth noting that the GHOSH algorithm does not enforce this choice and other pdfs could be used. However, Kalman filter analysis equations somehow prescribe a Gaussian approximation, thus the GHOSH filter keeps a link to Gaussianity, even if the GHOSH sampling does not. Interestingly, other filters, like , try to overcome this limitation and could represent good candidates to study the effects of a non-Gaussian GHOSH sampling.

The scaling matrix Λi used in the sampling procedure plays an important role in the filter, since Λi defines how variables are compared to each other in the computation of the most relevant components of the state uncertainty (Eq. ). In the Lorenz96 case, it is straightforward to set Λi equal to the identity matrix, since each variable is equivalent to any other. In more complex applications, Λi should be carefully designed to focus on the variables relevant for the processes of interest see alsofor a non-identity Λi and further discussion.

In the present implementation, the GHOSH filter derives the uncertainty subspace basis (Eq. ) using a T matrix (Eq. ) inherited from the SEIK filter. This is an advantage from the computational point of view, but ensemble members after the resampling may be very different from the original ensemble. For this reason, in some applications issues can arise, for instance, in the case of the physical balance constraints. As a solution, propose a filter (the error subspace transform Kalman filter, ESTKF) that, during the analysis step, minimizes the unnecessary changes to the forecast ensemble, and such approach can also be applied to the GHOSH filter algorithm by choosing an appropriately designed T matrix. However, differently from the ESTKF case, the T matrix should change at every forecast to take into account GHOSH projections onto the principal components of the uncertainty subspace, introducing an additional complexity layer to the newly presented GHOSH filter. Thus, given the additional complexity of the ESTKF approach and considering that: (i) its effects are less important if the state vector does not strongly need to preserve specific nonlinear constrains; (ii) showed that, in Lorenz96, SEIK with random rotation (adopted also in GHOSH) is as good as ESTKF; in the present work we limited to adopting a GHOSH filter that relies on a SEIK-like approach.

Compared to other ensemble filters, the key feature of the GHOSH filter is the higher polynomial order of its sampling method. The advantage of a higher order is not limited to a better estimation of the mean state (Appendix ) but it extends to the covariance matrix of the uncertainty probability distribution. In fact, the polynomial order in the estimation of the covariance matrix is half the order of the mean (not shown). Since the covariance matrix is involved in the state estimation, the more accurate GHOSH results shown in Sect. can be partly related to an improvement in the approximation error affecting the uncertainty covariance matrix. Moreover, it is worth noting that a higher order of the sampling positively impacts the approximation of the nonlinearity of the model independently of its Gaussianity. The examples in Appendix C show how moments of order higher than two can affect a sampling of a Gaussian pdf. In fact, even if its first two moments completely characterize the distribution, the sampling needs to also match the higher moments to produce an ensemble capable of achieving an order of approximation higher than two.

Similarly to other ensemble filters, a weighted ensemble is used in the GHOSH filter. However, the GHOSH filter differs substantially from the particle filter e.g., and from other types of weighted ensemble filters, e.g., and . In the listed filters, weights (representing likelihood) change over time, while particles remain the same and strategies are applied to resample the particles when weights “collapse”, e.g.,. In the case of GHOSH, the ensemble is used to estimate specific moments and then resampled to keep the weights constant in time, because those weights have specific desired properties that help reduce the error of the mean estimation. In this, GHOSH has similarities with the nonlinear ensemble transform filter , which does a resampling to keep the weights constant but, differently from GHOSH, with uniform weights and a second-order sampling procedure.

Finally, the asymptotic computational complexity of the GHOSH filter (Sect. ) is comparable to second-order deterministic filters (e.g., SEIK, ETKF). In our experiments (Sect. ), GHOSH showed an increased computational time with respect to SEIK (on average, 4 % of the total computational time), mainly due to the GHOSH filter's eigenvalue decomposition. However, it is worth noting that the cost related to the eigenvalue decomposition only depends on the number of ensemble members, affecting the Or3 part of the full GHOSH complexity Or3+N+nr2. As such, the eigenvalue decomposition is an important term if N+n (i.e., the sum of the dimensions of model and observations) is small (in our experiments on the Lorenz96 model it is smaller than 100), but becomes orders of magnitude less relevant in a realistic geoscience application, where degrees of freedom can be of the order of the millions and above. Moreover, while the model integration time and the filter computational time are still comparable in our experiments (12 % and 16 % for SEIK and GHOSH respectively), in a realistic application this will not be the case. Indeed, the filters computational complexity shows that filter computational time grows linearly with model dimension and the same holds for the Lorenz96 model that, however, is orders of magnitude less complex and computationally expensive than a realistic geoscience model. This is consistent with results obtained in a companion paper , where the GHOSH filter computational time in a realistic application accounts for less than 1 % of the total computational cost.

In summary, the benefits of the higher order offered by the GHOSH filter rely on a mathematical advantage with limited impacts on computational effort. In fact, GHOSH allows the exploitation of the degrees of freedom in the randomness of the sampling process, choosing ensembles that have a better consistency with the uncertainty distribution.

This work introduces two novel ensemble strategies: a sampling method (the high-order sampling) and an ensemble hybrid DA filter (GHOSH). Exploiting the high-order sampling, the GHOSH filter features a higher order of approximation compared to other ensemble filters, which resulted in better data assimilation performance.

Based on the results of the present work, the order of an ensemble method can be one of the proxies for filter skill, as shown by comparing SEIK (order 2) with GHOSH (order higher than 2).

While increasing the order usually involves increasing the number of ensemble members (and consequently the computational cost), the GHOSH approach keeps the same computational complexity and number of ensemble members as SEIK or ETKF and exploits the advantage of a higher order only where the approximation errors are larger.

In this work the validation and implementation of the GHOSH filter is limited to the Lorenz96 idealized setting, while the feasibility in a realistic geophysical application has been assessed in a companion paper by testing a 3D parallel implementation of a Mediterranean physical-biogeochemical model.

The improvements of the GHOSH filter (in its non-hybrid version) with respect to the SEIK filter are demonstrated in an extensive set of twin experiments based on the Lorenz96 model. The GHOSH filter showed an improved capacity to take into account nonlinearities through a lower need for inflation with respect to SEIK, along with the capability of performing a successful assimilation even when observation sparsity was causing SEIK to fail. In every configuration, the GHOSH filter proved to be as good as SEIK or better. Considering each filter tuned with its optimal forgetting factor, the GHOSH filter significantly improved the assimilation skill with respect to SEIK, achieving its best RMSE reduction (56 %) in the setting with moderate ensemble size (31 members).