To introduce the NICAS method, we start with a brief order-of-magnitude analysis of the computational cost of the application of a correlation matrix to a vector. Let us assume that a domain of dimensionality k (e.g. k=2 on a plane or on the sphere) is discretized into a grid G containing n cells of typical size γ and typical volume proportional to γk. Since the total volume of the domain V∝nγk is fixed, we can deduce that n∝γ-k. Applying a dense correlation operator C∈Rn×n to a vector x∈Rn requires K∝n2 operations, which is not computationally affordable for large systems (e.g. n∼109 for NWP operational systems).

However, as illustrated on the plane (k=2) in Fig. a, if the correlation function defining C is compactly supported with a typical support radius r, the number of points within the support is proportional to rk≪V and C becomes sparse. The number of operations K can be be significantly reduced: K∝rkn instead of n2. Introducing the correlation resolution ρ=r/γ, i.e. the correlation support radius r expressed as a function of the grid cell size γ, we finally get K∝ρk. In many cases, ρ is rather large. For instance, a 300 km correlation support radius for a 10 km resolution grid gives a correlation resolution ρ=30.

The fundamental idea of the NICAS method, as illustrated in Fig. b, is to apply the correlation operator on a dedicated subgrid G^ of size n^ and typical cell size γ^, for which the correlation resolution ρ^=r/γ^ is minimal, in order to reduce the number of operations from K∝ρk to K^∝ρ^k. However, ρ^ should be large enough to keep the correlation function sharpness on G^. To define this adaptive subgrid G^, we start from the correlation resolution ρ^ prescribed by the user, from which we deduce the typical subgrid cell size γ^=r/ρ^.

The NICAS correlation operator C on the grid G is given by 1C=NSC^STNT where C^∈Rn^×n^ is the correlation operator on G^, S∈Rn×n^ is an interpolation from G^ to G, N∈Rn×n is a diagonal operator ensuring that C is normalized (i.e. Cii=1), because even if C^ is normalized, SC^ST is not normalized in general.

If ρ is large enough, then ρ^ can be set a value significantly smaller than ρ, in which case the cost of the NICAS method should become dominated by the interpolation S, which grows linearly with n. Another important feature of the NICAS approach is the large flexibility given to C^, which is fully explicit. Thus, it can be inhomogeneous and anisotropic, non-separable, and handle complex boundaries. A final key aspect is the subgrid adaptiveness: the local density of G^ should be a function of the local correlation support radius of C^.

Since C^ is a correlation operator, it is symmetric positive definite. Thus, there is an infinity of possible square-roots U^∈Rn^×m such that C^=U^U^T. The number m of columns of U^ depends on the chosen modeling for U^. Once U^ is defined, it is straightforward to provide a square-root of C denoted U∈Rn×m: 2U=NSU^. In practice, it is always better to implement U and UT than C directly, for at least two reasons:

ensuring the positive definiteness of the implementation of C can be numerically challenging, while UUT is always symmetric positive (semi-)definite by construction,

The shape of the correlation function in C^ is completely free in the NICAS method, as long as its square-root is known. However, we have chosen to stick to the widely used piecewise polynomial, compactly supported function from , denoted GC99 hereafter. For sake of simplicity, a slightly different scaling is used in this function compared to the original paper (Sect. 4c), to ensure that it goes to zero when the input argument goes to one: 3C(d)=1-8d5+8d4+5d3-203d2ifd≤1283d5-8d4+5d3+203d2-10d+4-13d-1if12<d≤10if1<d. Here, the normalized distance d corresponds in the original paper to |z|/2c where |z| is the distance to the origin and c is the half support radius. As explained in the original paper, this function is obtained “by self-convolving the continuous, piecewise linear function”, which provides an explicit square-root function U(d). Within the more general parametric family developed in the original paper, the special case of C(d) corresponds to a linear square-root function U(d): 4U(d)=1-2difd≤120if12<d. In the discrete case, the correlation square-root U^ is built as: 5U^ij=Ni′Udijwithdij=Z(i,j)r where Z(i,j) is a measure of the distance between the points of G^ corresponding to indices i and j, and Ni′ is an internal normalization factor ensuring that C^ii=1. Figure  illustrates this correlation function on a regular 1D subgrid, with a correlation resolution ρ^=8:

A vector δ of impulses is defined on G^ (δi=0 except for a few elements where δi=1).

If the domain dimensionality k is larger than 1, the correlation operator can be made anisotropic by replacing the normalized distance dij in Eq. () with a tensor-based normalized distance: 6dij=(x(i)-x(j))TD-1(x(i)-x(j)) where x(i)∈Rk is the vector of coordinates of the point of G^ corresponding to index i, D∈Rk×k is a support tensor. In , the local correlation tensor is described as “a natural generalization of the (square of) the Daley length-scale for characterizing the spatial scales of the function”. Similarly here, the support tensor D is a natural generalization of the support radius r of the GC99 function. The components of D define the support of the function as a k-dimensional ellipsoid. Figure  shows examples of the GC99 function with different support tensors in a 2D case.

To change the shape of the correlation function, two strategies are possible:

either using another correlation function for C(d), but a square-root U(d) must be explicitly available,

The correlation operator can also can be made inhomogeneous by using a locally varying support radius r or support tensor D. If the subgrid is kept regular, the sharpness of the correlation operator becomes inhomogeneous as well, as illustrated by Fig. . For the left-hand side impulse of Fig. , where the support radius r is small, the local correlation resolution ρ^ is lower than 5, whereas it is larger than 10 for the right-side impulse where the support radius r is large. To enforce a homogeneous resolution ρ^=r/γ^ when the support radius r is locally varying, the only solution is to introduce a locally varying cell size γ^ following the same pattern. Figure  shows such a case: the cell size is smaller on the left-hand side where r is small, and larger on the right-hand side where r is large. Thus, the sharpness of the correlation function is homogeneous, whatever the local support radius. It should be noted that the internal normalization N′ becomes inhomogeneous for an inhomogeneous correlation resolution ρ^, but tends to be homogeneous when the cell size is adjusted to keep a homogeneous ρ^ (not shown in figures).

In theory, the normalized distance dij defined in Eqs. () and () could be fully tridimensional. However, in stratified fluids like the atmosphere or the ocean, it makes sense to split the normalized distance into horizontal and vertical components. As a consequence, the tensor formulation of Eq. () is always used in practice, but with zero cross-components between the horizontal and the vertical directions: 7D=D1Doff0DoffD2000rv2 where D1, D2 and Doff are the horizontal support tensor components, and rv is the vertical support radius. In the horizontally isotropic case: D1=D2=rh2 and Doff=0, where rh is the horizontal support radius. In the horizontally anisotropic case, Eq. (14) of defines an equivalent horizontal support radius rh associated with the horizontal part of the support tensor. rh is chosen so that the area of the ellipse defined by horizontal support tensor is equal to the area of the circle of radius rh: 8rh=D1D2-Doff21/4. This equivalent support radius rh will be useful to generate the adaptive horizontal subgrid.

Theoretically, G^ could be any tridimensional structured or unstructured subgrid. However, it is also relevant here to separate the vertical and horizontal directions, introducing an intermediate grid G̃ as illustrated on Fig. :

First, a vertical sub-sampling of G is performed, based on the vertical support radius rv and the resolution ρ^: only a subset of the levels of the G are kept to define the intermediate grid G̃.

While the obvious choice for the horizontal distance measure on the sphere is the great-circle distance, the choice of vertical coordinate is completely free in NICAS. Any vertical coordinate field provided on G can be used, as long as the vertical support radius rv has the same unit as the provided coordinate. This feature is very convenient for modeling vertically coupled correlations between Earth components, like atmosphere and ocean.

Different strategies are available to generate the horizontal subgrid at each selected level of the intermediate grid G̃: a.

If the local correlation radius r is homogeneous, a regular subgrid with a known (quasi-)homogeneous resolution is relevant. For a global domain on the sphere, an octahedral Gaussian grid is a good choice. Even if its cell size is slightly inhomogeneous (larger at mid-latitudes, smaller at the poles), it is possible to precisely select its truncation to obtain the desired subgrid cell size γ^ on average. For a regional domain, a subgrid made of equilateral triangles on the projection plane is a good candidate.

The implementation of the interpolation S in Eq. () should depend on the usage of the NICAS method. For the vertical direction, a linear interpolation seems to be sufficient, even if a higher order one could be easily implemented. For the horizontal direction, assuming that G^ is unstructured, the simplest option is a linear interpolation based on the Delaunay tesselation of the subgrid. However, the output field resulting from this type of interpolation is of class C0, which means that it is continuous but has potentially discontinuous derivatives. This discontinuity can be harmful if the derivatives of the analysis variables are important for subsequent calculations. For instance, if the analysis wind variables are the stream function φ and the velocity potential ψ, the physical wind components u and v based on the derivatives of φ and ψ will be discontinuous, as shown in the left panel of Fig. .

A possible solution is to introduce a more advanced interpolation producing fields of class C1, with continuous first-order derivatives. The SSRFPACK library implements such an interpolation, using local estimates of the function gradient at each subgrid point . Even if this interpolation significantly improves the continuity of u and v in the central panel of Fig. , the results are still noisy because of the errors in the local gradient estimation procedure provided with SSRFPACK.

Since the convolution operator C^ on G^ is a smoother, there is actually no need for an accurate interpolation between G^ and G. A smoothing interpolation could ensure smooth results on G. Its local smoothing support radius should be defined as a fraction of the horizontal correlation support radius rh, for consistency. However, this secondary smoothing would increase the initially intended smoothing on G. To alleviate this issue, an empirical reduction factor can be applied on the horizontal support radius (or tensor), which in turn increases the size of G^ and the overall cost. So this option is slightly more costly, but it provides very clean results as shown on the right panel of Fig. . While the differences seem minor between the correlation functions themselves for the three different interpolations (top row), the impact on the derivative can very large (bottom row).

One significant advantage of the NICAS method over other existing methods is its exact normalization (Cii=1). Once the square-root of the correlation operator on G^ (i.e. U^) and the interpolation from G^ to G (i.e. S) are defined, the normalization operator N is computed as the inverse of the square-root of the diagonal of SU^U^ST. Defining the vector δi∈Rn as a vector of zeros, except for a value one at the ith position, Nii can be computed as: 9Nii=δiTSU^U^STδi-1/2=U^STδi,U^STδi-1/2=U^STδi2-1 where 〈⋅〉 denotes a canonical inner product and ∥⋅∥2 the corresponding L2 norm. Thus, Nii can be exactly computed using Algorithm 1. This operation involves a lot of bookkeeping and it can be costly, but a parallel implementation is straightforward and significantly reduces this cost.

To illustrate how the NICAS method works in practice, the operator C of Eq. () is applied to an impulse vector δi, and the result at each step is displayed. Figure  shows these steps for an isotropic correlation function, and Fig.  for an anisotropic correlation function with a land/sea mask. The normalization N ensures that the correlation function maximum is equal to one at the end of the last step, even if the impulse on G is not collocated with a point G^. To take the land/sea mask into account in Fig. :

all the masked points (land mask) are removed from both G and G^,

All the experiments in this section are performed on a single horizontal level, because the vertical aspect of NICAS has little impact on performances. The grid G is an octahedral Gaussian grid at truncation TCo600 (n=1461600, γ∼16.5 km at the equator) and is the same for all the experiments. To obtain a given correlation resolution ρ on G, the correlation support radius r is given by r=γρ. Then, to obtain a given correlation resolution ρ^ on G^, the subgrid cell size is set as γ^=r/ρ^=γρ/ρ^. As a consequence, the subgrid size is n^∝γ^-2∝ρ^/ρ2. Since r is homogeneous, G^ is defined as a regular octahedral Gaussian grid with the appropriate truncation to fit the target subgrid cell size γ^, unless stated otherwise. The interpolation method is the simplest C0 linear interpolation based on a Delaunay tesselation of G^, unless stated otherwise. All the experiments are performed on a dedicated compute node of the BullSequana XH2000 HPC of ECMWF. The results are the average elapsed timings of 10 executions for the setup step (where the NICAS operators are computed) and 100 executions for the application step (where they are applied to a vector).

As explained in Sect. 2.1, the correlation resolutions ρ and ρ^ have a major impact on the cost of the NICAS method, and they can make it very competitive or very costly. An order-of-magnitude analysis can be conducted for the different steps of the NICAS method:

For the setup step:

The cost of the subgrid generation is mostly due to the assignment of each subgrid point to a given MPI task, depending on its location. This cost increases linearly with the subgrid size n^∝ρ^/ρ2.

Figure  provides a confirmation of this analysis, for experiments on 32 MPI tasks, with a few more interesting points. First, the setup step is several orders of magnitude more expensive than the application step. In the application step, the cost of communications is negligible (only local halo exchange), the cost of normalization is also very small, and the cost of convolution is significantly smaller than the cost of interpolation. This was the fundamental idea of the NICAS method design detailed in Sect. 2.1: making the whole cost dominated by the interpolation, which depends on the size n of G and the type of interpolation, independently from the convolution itself.

Figure  shows the impact of the interpolation type alone, the other parameters being held constant (32 MPI tasks, ρ=20, ρ^=8). Among the three interpolations tested here, the C0 linear interpolation based on a Delaunay tesselation is of course the cheapest. The C1 upgrade, which requires the estimation of gradients at the vertices of the interpolation triangle is slightly more expensive. With the smoothing interpolation, needed to get continuous derivatives, the correlation support radius r needs to be artificially decreased in order to compensate for the “double-smoothing” in both the convolution and the interpolation operators. In the current experiment, r needed to be multiplied by a factor 0.7 to obtain the same total correlation radius. Thus, the size n^ of the subgrid was multiplied by approximately 0.7-2≃2, which explains the amplitude of the cost increase for both the setup and the application steps.

To evaluate the NICAS method scalability, the same experiment (ρ=20, ρ^=8) is run with either 16, 32, 64 or 128 MPI tasks. Since a single HPC node is always used, the cost of communications might be underestimated here. Indeed, it would probably be larger on several nodes because of slower inter-nodes communications. For sake of comparison, a “perfect scalability” curve is computed by extrapolating the timing of the case with the lowest number of MPI tasks and dividing this timing by a factor 2 each time the number of MPI tasks increases by a factor 2.

Figure  shows that the setup step does not scale perfectly, mostly because of the interpolation setup and the subgrid generation. Indeed, the tesselation algorithm (STRIPACK) that accounts for an important part of the interpolation setup is not parallel. Similarly, the assignment of each subgrid point to a given MPI task in the subgrid generation is not parallel neither. However, the application step scales very well and closely follows the “perfect scalability” curve. This is a real important feature of the NICAS method.

As noted before, the setup step is several orders of magnitude more expensive than the application step and it does not scale as well, even though most of the development efforts have been (and still are) devoted to its acceleration. But it is not a blocking issue because:

The setup step can be run before the time critical path of the data assimilation process, i.e. as a preparation step before all the observations are available.