We first construct a likelihood function that models the observations by comparing certain summary statistics of the observations to the corresponding statistics of a trajectory simulated from the chaotic model. As a source of statistics, we will choose the correlation integral, which depends on the fractal dimension of the chaotic attractor. Unlike other statistics – such as the ergodic averages of a trajectory – the correlation integral is able to constrain the parameters of a chaotic model .

Let us denote by 1dudt=f(u,θ),u(t=0)=u0, a dynamical system with state u(t)∈Rn, initial state u0∈Rn, and parameters θ∈Rq. The time-discretized system, with time steps ti∈{t1,…,tτ} denoting selected observation points, can be written as 2ui≡u(ti)=F(ti;u0,θ). Either the full state ui∈Rn or a subset si∈Rd≤n of the state components are observed. We will use S={s1,…,sτ} to denote a collection of these observables at successive times.

Using the model–observation mismatch at a collection of times ti to constrain the value of the parameters θ is not suitable when the system () has chaotic dynamics, since the state vector values si are unpredictable after a finite time interval. Though long-time trajectories s(t) of chaotic systems are not predictable in the time domain, they do, however, represent samples from an underlying attractor in the phase space. The states are generated deterministically, but the model's chaotic nature allows us to interpret the states as samples from a particular θ-dependent distribution. Yet obvious choices for summary statistics T that depend on the observed states S, such as ergodic averages, ignore important aspects of the dynamics and are thus unable to constrain the model parameters. For example, the statistic T(S)=1τ∑i=1τsi is easy to compute and is normally distributed in the limit τ→∞ (under appropriate conditions), but this ergodic mean says very little about the shape of the chaotic attractor.

Instead, we need a summary statistic that retains information relevant for parameter estimation but still defines a computationally tractable likelihood. To this end, devised the CIL, which retains enough information about the attractor to constrain the model parameters. We first review the CIL and then discuss how to make evaluation of the likelihood tractable.

We will use the CIL to evaluate the “difference” between two chaotic attractors. For this purpose, we will first describe how to statistically characterize the geometry of a given attractor, given suitable observations S. In particular, constructing the CIL likelihood will require three steps: (i) computing distances between observables sampled from a given attractor; (ii) evaluating the empirical cumulative distribution function (ECDF) of these distances and deriving certain summary statistics T from the ECDF; and (iii) estimating the mean and covariance of T by repeating steps (i) and (ii).

Intuitively, the CIL thus interprets observations of a chaotic trajectory as samples from a fixed distribution over phase space. It allows the time between observations to be arbitrarily large – importantly, much longer than the system's non-chaotic prediction window.

Now we describe the CIL construction in detail. Suppose that we have collected a data set S comprising observations of the dynamical system of interest. Let S be split into nepo different subsets called epochs. The epochs can, in principle, be any subsets of length N from the reference data set S. In this paper, however, we restrict the epochs to be time-consecutive intervals of N evenly spaced observations. Let sk={sik}i=1N and sl={sjl}j=1N, with 1≤k,l≤nepo and k≠l, be two such disjoint epochs. The individual observable vectors sik∈Rd and sjl∈Rd comprising each epoch come from time intervals [tkN+1,t(k+1)N] and [tlN+1,t(l+1)N], respectively. In other words, superscripts refer to different epochs and subscripts refer to the time points within those epochs. then define the modified correlation integral sum C(R,N,sk,sl) by counting all pairs of observations that are less than a distance R>0 from each other: 3C(R,N,sk,sl)=1N2∑i,j≤N1[0,R]sik-sjl, where 1 denotes the indicator function and ∥⋅∥ is the Euclidean norm on Rd. In the physics literature, evaluating Eq. () in the limit R→0, with k=l and i≠j, numerically approximates the fractal dimension of the attractor that produced sk=sl . Here, we instead use Eq. () to characterize the distribution of distances between sk and sl at all relevant scales. We assume that the state space is bounded; therefore, an R0 covering all pairwise distances in Eq. () exists. For a prescribed set of radii Rm=R0b-m, with b>1 and m=0,…,M, Eq. () defines a discretization of the ECDF of the distances ∥sik-sjl∥, with discretization boundaries given by the numbers Rm.

Now we define ymk,l=C(Rm,N,sk,sl) as components of a statistic T(sk,sl)=yk,l:=(y0k,l,…,yMk,l). This statistic is also called the feature vector. According to and , the vectors yk,l are normally distributed, and the estimates of the mean and covariance converge at the rate nepo to their limit points. This is a generalization of the classical result of , which applies to Independent and identically distributed samples from a scalar-valued distribution. We characterize this normal distribution by subsampling the full data set S. Specifically, we approximate the mean μ and covariance Σ of T by the sample mean and sample covariance of the set {yk,l:1≤k,l≤nepo,k≠l}, evaluated for all 12nepo(nepo-1) pairs of epochs (sk,sl) using fixed values of R0, b, M, and N.

The Gaussian distribution of T effectively characterizes the geometry of the attractor represented in the data set S. Now we wish to use this distribution to infer the parameters θ. Given a candidate parameter value θ̃, we use the model to generate states s*(θ̃)={si*(θ̃)}i=1N for the length of a single epoch. We then evaluate the statistics ymk,*=C(Rm,N,sk,s*(θ̃)) as in Eq. (), by computing the distances between elements of s*(θ̃) and the states of an epoch sk selected from the data S. Combining these statistics into a feature vector yk,*(θ̃)=(ymk,*)m=0M, we can write a noisy estimate of the log-likelihood function: 4log⁡p(θ̃|sk)=-12yk,*(θ̃)-μ⊤Σ-1yk,*(θ̃)-μ+constant. Comparing s*(θ̃) with other epochs drawn from the data set S, however, will produce different realizations of the feature vector. We thus average the resulting log likelihoods over all epochs: 5log⁡p(θ̃|S)=1nepo∑k=1nepolog⁡p(θ̃|sk). This averaging, which involves evaluating Eq. () nepo times, involves only new distance computations and is thus relatively cheap relative to time integration of the dynamical model.

Because the feature vectors yk,* are random for any finite N, and because the number of epochs nepo is also finite, the log likelihood in Eq. () is necessarily random. It is then useful to view Eq. () as estimate of an underlying true log likelihood. We are therefore in a setting where cannot evaluate the unnormalized posterior density exactly; we only have access to a noisy approximation of it. Previous work has demonstrated that derivative-free optimizers such as the differential evolution (DE) algorithm can successfully identify the posterior mode in this setting, yielding a point estimate of θ. In the fully Bayesian setting, one could characterize the posterior p(θ|S) using pseudo-marginal MCMC methods but at significant computational expense. Below, we will use a surrogate model constructed adaptively during MCMC sampling to reduce this computational burden.

Note that the CIL approach described above already reduces the computational cost of inference by only requiring simulation of the (potentially expensive) chaotic model for a single epoch. We compare each epoch of the data to the same single-epoch model output. Each of these comparisons results in an estimate of the log likelihood, which we then average over data epochs. A larger data set S can reduce the variance of this average, but does not require additional simulations of the dynamical model. Also, we do not require any knowledge about the initial conditions of the model; we omit an initial time interval before extracting s*(θ̃) to ensure that the observed trajectory is on the chaotic attractor.

Moreover, the initial values are randomized for all simulations and sampling is started only after the model has integrated beyond the initial, predictable, time window. The independence of the sampled parameter posteriors from the initial values was verified both here and in earlier works by repeated experiments.

Our approach is broadly similar to the synthetic likelihood method (e.g., ) but differs in two key respects: (i) we use a novel summary statistic that is able to characterize chaotic attractors, and (ii) we only need to evaluate the forward model for a single epoch. Comparatively, synthetic likelihoods typically use summary statistics such as auto-covariances at a given lag or regression coefficients. These methods also require long-time integration of the forward model for each candidate parameter value θ, rather than integration for only one epoch. also discuss several ways of using feature vectors for inference in geophysics. A distinction of the present work is that we use an ECDF-based summary statistic that is provably Gaussian, and we perform extensive Bayesian analysis of the parameter posteriors via novel MCMC methods. These methods are described next.

Even with the developments described above, estimating the CIL at each candidate parameter value θ̃ is computationally intensive. We thus use local approximation MCMC (LA-MCMC) – a surrogate modeling method that replaces many of these CIL evaluations with an inexpensive approximation. Replacing expensive density evaluations with a surrogate was first introduced by and . LA-MCMC extends these ideas by continually refining the surrogate during sampling, which guarantees convergence.

First introduced in , LA-MCMC builds local surrogate models for the log likelihood while simultaneously sampling the posterior. The surrogate is incrementally and infinitely refined during sampling and thus tailored to the problem – i.e., made more accurate in regions of high posterior probability. Specifically, the surrogate model is a local polynomial computed by fitting nearby evaluations of the “true” log likelihood. We emphasize that the approximation itself is not locally supported. At each point θ̃, we locally construct a polynomial approximation, which globally defines a piecewise polynomial surrogate model. This is an important distinction because the piecewise polynomial approximation is not necessarily a probability density function. In fact, the surrogate function may not even be integrable. Despite this challenge, devise a refinement strategy that ensures convergence and bounds the error after a finite number of samples. In particular, shows that the error in the approximate Markov chain computed with the local surrogate model decays at approximately the expected 1/T rate, where T is the number of MCMC steps. demonstrated that noisy estimates of the likelihood are sufficient to construct the surrogate model and still retain asymptotic convergence. Empirical studies on problems of moderate parameter dimension showed that the number of expensive likelihood evaluations per MCMC step can be reduced by orders of magnitude, with no discernable loss of accuracy in posterior expectations.

Here, we briefly summarize one step of the LA-MCMC construction and refer to for details. Each LA-MCMC step consists of four stages: (i) possibly refine the local polynomial approximation of the log likelihood, (ii) propose a new candidate MCMC state, (iii) compute the acceptance probability, and (iv) accept or reject the proposed state. The major distinction between this algorithm and standard Metropolis–Hastings MCMC is that the acceptance probability in stage (iii) is computed only using the approximation or surrogate model of the log likelihood, at both the current and proposed states. This introduces an error, relative to computation of the acceptance probability with exact likelihood evaluations, but stage (i) of the algorithm is designed to control and incrementally reduce this error at the appropriate rate.

“Refinement” in stage (i) consists of adding a computationally intensive log-likelihood evaluation at some parameter value θi, denoted by L(θi), to the evaluated set {(θi,L(θi))}i=1K. These K pairs are used to construct the local approximation via a kernel-weighted local polynomial regression . The values {θi}i=1K are called “support points” in this paper. Details on the regression formulation are in . As the support points cover the regions of high posterior probability more densely, the accuracy of the local polynomial surrogate will increase. This error is well understood and, crucially, takes advantage of smoothness in the underlying true log-likelihood function. This smoothness ultimately allows the cardinality of the evaluated set to be much smaller than the number of MCMC steps.

Intuitively, if the surrogate converges to the true log likelihood, then the samples generated with LA-MCMC will (asymptotically) be drawn from the true posterior distribution. After any finite number of steps, however, the surrogate error introduces a bias into the sampling algorithm. The refinement strategy must therefore ensure that this bias is not the dominant source of error. At the same time, refinements must occur infrequently to ensure that LA-MCMC is computationally cheaper than using the true log likelihood. analyzes the trade-off between surrogate-induced bias and MCMC variance and proposes a rate-optimal refinement strategy. We use a similar algorithm, only adding an isotropic ℓ2 penalty on the polynomial coefficients. More specifically, we rescale the variables so the max/min values are ±1 – often called coded units – then regularize the regression by a penalty parameter (in our cases, the value α=1 was found sufficient). This penalty term modifies the ordinary least squares problem into local ridge regression, which improves performance with noisy likelihoods.

Our examples use an adaptive proposal density . This choice deviates slightly from the theory in , which assumes a constant-in-time proposal density. However, this does not necessarily imply that adaptive or gradient-based methods will not converge. In particular, show asymptotic convergence using an adaptive proposal density and strengthen this result by showing that the Metropolis-adjusted Langevin algorithm, which is a gradient based MCMC method, is asymptotically exact when using a continually refined local polynomial approximation. These results require some additional assumptions about the target density's tail behavior and the stronger rate optimal result from has not been shown for such algorithms. However, in practice, we see that adaptive methods still work well in our applications. Exploring the theoretical implications of this is interesting and merits further discussion but is beyond the scope of this paper.

The parameters of the algorithm are fixed as given in , for all the examples discussed here: (i) initial error threshold γ0=1; (ii) error threshold decay rate γ1=1; (iii) maximum poisedness constant Λ¯=100; (iv) tail-correction parameter η=0 (no tail correction); (v) local polynomial degree p=2. The number of nearest neighbors k used to construct each local polynomial surrogate is chosen to be k=k0+(K-k0)1/3 where k0=qD, q is the dimension of the parameters θ∈Rq, and D is the number of coefficients in the local polynomial approximation of total degree p=2, i.e., D=(q+2)(q+1)/2. If we had k=D, the approximation would be an interpolant. Instead, we oversample by a factor q, as suggested in , and allow k to grow slowly with the size K of the evaluated set as in . All these details together with example runs can be found in the MATLAB implementation available in the Supplement.

We use the classical three-dimensional Lorenz 63 system as a simple first example to demonstrate how LA-MCMC can be successfully paired with the CIL and the adaptive Metropolis (AM) algorithm to obtain the posterior distribution for chaotic systems at a greatly reduced computational cost, compared to AM without the local approximation. The time evolution of the state vector s=(X,Y,Z) is given by 8X˙=σ(Y-X),Y˙=X(ρ-Z)-Y,Z˙=XY-βZ. This system of equations is often said to describe an extreme simplification of a weather model.

The reference data were generated with parameter values σ=10, ρ=28, and β=83 by performing nepo=64 distinct model simulations, with observations made at 2000 evenly distributed times between [10,20000]. These observations were perturbed with 5 % multiplicative Gaussian noise. The length of the predictable time window is roughly 7, which is less than the time between consecutive observations. The parameters of the CIL method were obtained as described at the start of Sect. , with values M=14, R0=2.85, and b=1.51.

The set of vectors {yk,l|k,l≤nepo} is shown in Fig.  in the log–log scale. The figure shows how the variability of these vectors is quite small. Figure  validates the normality assumption for feature vectors.

Pairwise two-dimensional marginals of the parameter posterior are shown in Fig. , both from sampling the posterior with full forward model simulations (AM) and with using the surrogate sampling approach for generating the chain. These two posteriors are almost perfectly superimposed. Indeed, the difference is at the same level as that between repetitions of the standard AM sampling alone.

To get an idea of the computational savings achieved with LA-MCMC, the computation of the MCMC chains of length 105 was repeated 10 times. The cumulative number of full likelihood evaluations is presented in Fig. . At the end of the chains, the number of full likelihood evaluations varied between 955 and 1016. Thus, by using LA-MCMC in this setting, remarkable computational savings of up to 2 orders of magnitude are achieved.

The second example is the 256-dimensional Kuramoto–Sivashinsky (KS) partial differential equation (PDE) system. The purpose of this example is to introduce ways to improve the computational efficiency by a piecewise parallel integration over the time interval of given data. Also, we demonstrate how decreasing the number of observed components impacts the accuracy of parameter estimation. Even though the posterior evaluation proves to be relatively expensive, direct verification of the results with those obtained by using standard adaptive MCMC is still possible. The Kuramoto–Sivashinsky model is given by the fourth-order PDE: 9st=-ssx-1ηsxx-γsxxxx, where s=s(x,t) is a real function of x∈R and t∈R+. In addition, it is assumed that s is spatially periodical with period of L, i.e., s(x+L,t)=s(x,t). This experiment uses the parametrization from that maps the spatial domain [-L2,L2] to [-π,π] by setting x̃=2πLx and t̃=2πL2t. With L=100, the true value of parameter γ is (π/50)2≈0.0039, and the true value of η becomes 12. These two parameters are the ones that are then estimated with the LA-MCMC method. This system was derived by and as a model for phase turbulence in reaction–diffusion systems. used the same system for describing instabilities of laminar flames.

Assume that the solution for this problem can be represented by a truncation of the Fourier series 10s(x,t)=∑j=0∞Aj(t)sin⁡2πLjx+Bj(t)cos⁡2πLjx. Using this form reduces Eq. () to a system of ordinary differential equations for the unknown coefficients Aj(t) and Bj(t), 11A˙j(t)=α1j2Aj(t)+α2j4Aj(t)+F1(A(t))12B˙j(t)=β1j2Bj(t)+β2j4Bj(t)+F2(B(t)), where the terms F1(⋅) and F2(⋅) are polynomials of the vectors A and B. For details, see . The solution can be effectively computed on graphics processors in parallel, and if computational resources allow, several instances of Eq. () can be solved in parallel. Even on fast consumer-level laptops, several thousand simulations can be performed in parallel when the discretization of the x dimension contains around 500 points.

A total of 64 epochs of the 256-dimensional KS model are integrated over the time interval [0,150000], and as in the case of the Lorenz 63 model, the initial predictable time window is discarded and 1024 equidistant measurements from [500,150000] are selected, with Δt≈146. The parameters used for the CIL method were R0=1801.7, M=32, and b=1.025.

The time needed to integrate the model up to t=150000 is approximately 103 s with the Nvidia 1070 GPU, implying that generating an MCMC chain with 100 000 samples with standard MCMC algorithms would take almost 4 months. The use of LA-MCMC alone again shortens the time needed by a factor of 100 to around 28 h. However, the calculations can yet be considerably enhanced by parallel computing. In practice, this translates the problem of generating a candidate trajectory of length 150 000 into generating observations from several shorter time intervals. In our example, an efficient division is to perform 128 parallel calculations each of length 4500, with randomized initial values close to the values selected from the training set. Discarding the predictable interval [0,500] and taking eight observations at intervals of 500 yields the same number (1024) of observations as in the initial setting. While the total integration time increases, this reduces the wall-clock time needed for computation of a single candidate simulation from 103 to 2.5 s. The full MCMC chain can be then be generated in 70 h without the surrogate model and in 42 min using LA-MCMC.

Parameter posterior distributions from the KS system, produced with MCMC both with and without the local approximation surrogate, are shown in Fig. . Repeating the calculations several times yielded no meaningful differences in the results. In this experiment, the number of forward model evaluations LA-MCMC needed for generating a chain of length 100 000 was in the range [1131,1221].

Model trajectories from simulations with four different parameter vectors are shown in Fig. . These parameter values were (1) the “true” value which was used to generate training data, (2) another parameter from inside the posterior distribution, and (3–4) two other parameters from outside the posterior distribution. These parameters are also shown in Fig. . Visually inspecting the outputs, cases 1–3 look similar, while results using parameter vector 4, furthest away from the posterior, are markedly different. Even though the third parameter vector is outside the posterior, the resulting trajectory is not easily distinguishable from cases 1 and 2, indicating that the CIL method differentiates between the trajectories more efficiently.

Additional experiments were performed to evaluate the stability of the method when not all of the model states were observed. Keeping the setup otherwise fixed, the number of elements of the state vectors observed was reduced from the full 256 step by step to 128, 64, and 32. The resulting MCMC chains are presented in Fig. , and as expected, when less is observed, the size of the posterior distribution grows.

The methodology is here applied to a computationally intensive model, where a brute-force parameter posterior estimation would be too time consuming. We employ the well-known quasi-geostrophic model using a dense grid to achieve complex chaotic dynamics in high dimensions. The wall-clock time for one long-time forward model simulation is roughly 10 min, so a naïve calculation of a posterior sample of size 100 000 would take around 2 years. We demonstrate how the application of the methods verified in the two previous examples reduces this time to a few hours.

The QG model approximates the behavior on a latitudinal “stripe” at two given atmospheric heights, projected onto a two-layered cylinder. The model geometry implies periodic boundary conditions, seamlessly stitching together the extreme eastern and western parts of the rectangular spatial domain with coordinates x and y. For the northern and southern edges, user-specified time-independent Dirichlet boundary conditions are used. In addition to these conditions and the topographic constraints, the model parameters include the mean thicknesses of the two interacting atmospheric layers, denoted by H1 and H2. The QG model also accounts for the Coriolis force. An example of the two-layer geometry is presented in Fig. .

In a non-dimensional form, the QG system can be written as 13q1=Δψ1-F1(ψ1-ψ2)+βy,14q2=Δψ2-F2(ψ2-ψ1)+βy+Rs, where qi are potential vorticities, and ψi are stream functions with indexes i=1,2 for the upper and the lower layers, respectively. Both the qi and ψi are functions of time t and spatial coordinates x and y. The coefficients Fi=f02L2g´Hi control how much the model layers interact, and β=β0L/U gives a nondimensional version of β0, the northward gradient of the Coriolis force that gives rise to faster cyclonic flows closer to the poles. The Coriolis parameter is given by f0=2Wsin⁡(ℓ), where W is the angular speed of Earth and ℓ is the latitude of interest. L and U give the length and speed scales, respectively, and g´ is a gravity constant. Finally, Rs(x,y)=S(x,y)ηH2 defines the topography for the lower layer, where η=Uf0L is the Rossby number of the system. For further details, see and .

It is assumed that the motion determined by the model is geostrophic, essentially meaning that potential vorticity of the flow is preserved on both layers: 15∂qi∂t+ui∂qi∂x+vi∂qi∂y=0. Here, ui and vi are velocity fields, which are functions of both space and time. They are obtained from the stream functions ψi via 16ui=-∂ψi∂y,vi=∂ψi∂x. Equations ()–() define the spatiotemporal evolution of the quantities qi,ψi, i=1,2.

The numerical integration of this system is carried out using the semi-Lagrangian scheme, where the potential vorticities qi are computed according to Eq. () for given velocities ui and vi. With these qi the stream functions can then be obtained from Eqs. () and () with a two-stage finite difference scheme.

Finally, the velocity field is updated by Eq. () for the next iteration round.

For estimating model parameters from synthetic data, a reference data set is created with 64 epochs each containing N=1000 observations. These data are sampled from the model trajectory with Δt=8 (where a time step of length 1 corresponds to 6 h) in the time interval [192,8192], that amounts to a long-range integration of roughly 5–6 years of a climate model. The spatial domain is discretized into a 55×55 grid, which results in consistent chaotic behavior and more complex dynamics than with the often-used 20×40 grid. This is reflected in higher variability in the feature vectors, as seen in the Fig. . A snapshot of the 6050-dimensional trajectory of the QG system is displayed in Fig. .

The model state is characterized by two distinct fields, the vorticities and stream functions, that naturally are dependent on each other. As shown in , it is useful to construct separate feature vectors to characterize the dynamics in such situations. For this reason, two separate feature vectors are constructed – one for the potential vorticity on both layers and the other for the stream function.

The Gaussian likelihood of the state is created by stacking these two feature vectors one after another.

The normality of the resulting 2(M+1)-dimensional vector may again be verified as shown in Fig. . The number of bins was set to 32, leading to parameter values R0=55 and b=1.075 for potential vorticity, and R0=31, and b=1.046 for the stream function.

For parameter estimation, inferring the layer heights from synthetic data is considered. The reference data set with nepo=64 integrations is produced using the values H1=5500 and H2=4500. A single forward model evaluation takes 10 min on a fast laptop, and therefore generating MCMC chains of length 105 with brute force would take around 2 years to run. As previously, using LA-MCMC again reduces the computation time by a factor of 100.

In the experiments performed, the number of forward model evaluations needed was ranging in the interval [682,762], which translates to around 1 week of computing time. As verified with Kuramoto–Sivashinsky example, the forward model integration can be split to segments computed in parallel, which reduced time required to generate data for computing the likelihood further with a factor around 50, corresponding to around 3 h for generating the MCMC chain. The pairwise distances for generating the feature vectors were computed on a GPU, and therefore the required computation time for doing this was negligible compared to the model integration time. The posterior distribution of the two parameters is presented in Fig. .