Introduction
Mathematical models often contain roughly known model parameters which are
optimized based on measurements. The resulting accuracy of the model
parameters depends on the conditions, also called experimental setups or
experimental designs, under which these measurements are carried out. These
experimental designs can be optimized so that the resulting accuracy is
maximized. Thus, the effort and cost of measurements can be significantly
reduced.
The optimization of experimental designs is therefore particularly
interesting for geosciences, where much money is spent on data collection.
However, few application examples exist in this field (see
, for an overview). This article aims to promote this
approach in geosciences and exemplarily apply it to an existing salt marsh
accretion model ().
In optimizing experimental design, the main problem is to quantify the
information content of the measurements to be planned. In general, this can
only be done approximatively. There are several approaches available. In
Sect. , four different approaches to
optimize experimental designs together with the weighted least squares
estimator for model parameters are presented. Each of these four approaches
makes a different trade-off between accuracy and computational effort.
One approach is based on the linearization of the model with respect to the
parameters and is the most common used approach called local optimal
experimental design. The second more robust approach takes into account the
potential nonlinearity of the model parameters. Both approaches are combined
with two different approaches of solving the underlying discrete optimization
problem.
To the author's knowledge, there is no open-source software available that
can apply all four of these approaches. The only software using this robust
approach is a closed-source software called VPLAN which was introduced in
. For the local optimal approach, several implementations
are available, but there is no open-source software written in MATLAB. All
four approaches, together with approaches to optimize model parameters, were
implemented in a MATLAB toolbox called the Optimal Experimental Design Toolbox. Its structure and application is described in
Sect. .
We have chosen two models as application examples, simulating the suspended sediment concentration on
salt marshes during tidal inundation and resulting accretion rates on these
marshes
(). Both models are zero-dimensional point models and
differ in their complexity and number of parameters. These models can be used
as a basis to predict the survival capability of salt marshes under the
influence of expected global sea level rise.
To use these models for local salt marshes, their parameters have to be
adapted to the local environmental conditions. The required measurements are
very time-consuming and costly. Using the presented approaches, these
measurements could be carried out much more efficiently. These two models are
described together with the attendant numerical experiments and the
associated results in Sect. .
Optimization of model parameters and<?xmltex \hack{\break}?> experimental designs
The first step to the optimization of model parameters is the choice of the
estimator. This maps the measurement results onto estimated model parameters.
These estimated parameters are often defined so that they minimize a
so-called misfit function. The misfit function quantifies the distance
between the measurement results and the model output.
The estimator should be derived from the statistical properties of the
measurement errors, for example, a maximum likelihood estimator. Often the
measurement errors are assumed to be normally distributed; this leads to the
least squares estimators. They are the most widely used class of estimators
since their introduction by Gauss and Legendre ().
Their simplest form is the ordinary least squares estimator. Its misfit
function is the sum of the squares of the differences between each
measurement result and the corresponding model output. A generalization is
the weighted least squares estimator which has advantages in the event of
heteroscedastic measurement errors. This estimator and its asymptotic
properties are presented in the following subsection. The generalized least
squares estimator is a further generalization which takes into account the
stochastic dependence of the measurement errors.
The weighted least squares estimator
In the following, the weighted least squares estimator is presented. For this
purpose, some notations and assumptions are introduced.
The model function is denoted by
f:Ωx×Ωp→R.
Here, Ωx⊆Rnx is the set of feasible
experimental designs, and Ωp⊂Rnp is the set of
feasible model parameters from which the unknown exact parameter vector
p^∈Ωp is to be determined. Often these sets are
defined by lower and upper bounds.
The measurement result for every design x∈Ωx is considered
as a realization of a random variable ηx. Each random variable
ηx is assumed to be normally distributed with the expectation
f(x,p^) and standard deviation σx>0.
A1a.
ηx∼N(f(x,p^),σx2) for every x∈Ωx.
Furthermore, these random variables are assumed to be pairwise stochastically
independent.
A1b.
ηx and ηx′ are stochastically independent for every x,x′∈Ωx.
If we consider n≥np measurement results y=(y1,…,yn)T∈Rn with corresponding experimental
designs x1,…,xn∈Ωx, the weighted least
squares estimation pn and the corresponding estimator Pn are
defined as
pn:=Pn(y):=arg minp∈Ωpψn(y,p),
where the misfit function ψn is defined as
ψn:Rn×Ωp→R,(y,p)↦∑i=1nyi-f(xi,p)σxi2.
With the following assumptions, the existence of a minimum is ensured.
A2.
f(x,⋅) is continous for every x∈Ωx.
A3.
Ωp is compact (closed and bounded).
If ψn(y,⋅) is convex, the minimum is also unique.
The parameter estimation pn in
Eq. () can be calculated with
an optimization method for continuous optimization problems. A possible
method is the sequential quadratic programming (SQP) algorithm which is, for
example, described in chapter 18.
Asymptotic properties
Provided certain regularity conditions are met, the least squares estimators
are consistent, asymptotically normally distributed and asymptotically
efficient.
These asymptotic properties were first proved by for the
ordinary least squares estimator and also discussed in
and . In , these properties were proved for the
weighted least squares estimator and for the generalized least squares
estimator in . A good summary for all three can be
found in .
Consistency means that the estimated parameters converge in probability to
the unknown exact parameters as the number of measurements goes to infinity;
that is,
Pn⟶pp^asn→∞
for the weighted least squares estimator Pn with the unknown exact model
parameters p^.
For consistency, the following assumptions are sufficient in addition to the
previous assumptions A1 to A3 p. 565.
A4a.
n-1Bn converges uniformly with Bn:Ωp×Ωp→R,(p,p′)↦∑i=1nf(xi,p)f(xi,p′)σxi-2.
A4b.
D¯(p,p^)=0⇒p=p^ for all
p∈Ωp with D¯:=limn→∞n-1Dn and Dn:Ωp×Ωp→R,(p,p′)↦∑i=1n(f(xi,p)-f(xi,p′))2σxi-2 (D¯ is well defined by
assumption A4a).
An estimator is asymptotically efficient if its variance converges to the
Cramér–Rao bound as the number of measurements goes to infinity. The
Cramér–Rao bound () is a lower bound for the
variance of any unbiased estimator.
For asymptotic efficiency, the following assumptions are sufficient in
addition to the previous assumptions A1 to A4 p. 571.
A5.
p^ is an interior point of Ωp.
Let Ω^p⊆Ωp be an open neighborhood of
p^.
A6.
f(xi,⋅) is twice continuously differentiable
in Ω^p.
A7.
n-1Mn converges uniformly with Mn:Ω^p→Rnp×np,p↦∑i=1n∇pf(xi,p)∇pf(xi,p)Tσxi-2.
A8.
n-1Hn converges uniformly with Hn:Ω^p→Rnp×np,p↦∑i=1n(∂2∂pi∂pjf(xi,p))2σxi-2i,j=1,…,np.
A9.
M^(p^) is invertible with M^:=limn→∞n-1Mn.
In this case, the Cramér–Rao bound of the weighted least squares
estimator Pn is the inverse of the Fisher information matrix
Mn(p^).
Under these assumptions, the asymptotic behavior of the weighted least
squares estimator can be summarized by its convergence in distribution as
follows:
n(Pn-p^)⟶dN(0,Mn(p^)-1n) as n→∞
(see, e.g., chapter 12 and
chapter 3).
Optimal experimental designs
The accuracy of the weighted least square estimator Pn can be described by
its covariance matrix. Due to the asymptotic distribution
(Eq. ), this can be approximated by
the inverse of the information matrix Mn(pn), provided the matrix
Mn(pn) is nonsingular, that is,
cov(Pn)≈Mn(pn)-1.
Therefore, the unknown model parameters can be determined more accurately the
smaller the (approximated) covariance matrix of the estimator is.
Criteria ϕ:Rnp×np→R+∪{∞}, such as the trace or determinant, are used in order to compare
these matrices (see, e.g., , for an overview of various
criteria). If the approximation (Eq. )
is used and Mn(pn) is singular, the value of ϕ is set to
infinity.
In the context of optimizing experimental designs, we assume n≥0
measurements have been carried out and designs for additional measurements
should be selected from m designs x1′,…,xm′∈Ωx. The choice for each design xi′ is expressed by a weight
wi∈{0,1}, where 1 indicates the selection and 0 the contrary.
Hence, the resulting information matrix, depending on the choice w∈{0,1}m and the parameter vector pn∈Ωp, is defined as
Mn(w,pn):=Mn(pn)+∑i=1mwi∇pf(xi′,pn)∇pf(xi′,pn)Tσxi′2.
If the covariance matrix is approximated by the inverse of the information
matrix, optimal (additional) designs, with respect to a criterion ϕ, are
expressed by a solution of
arg minw∈{0,1}mϕ(Mn(w,pn)-1).
These optimal designs are called local optimal designs because these designs
are only optimal regarding the previous model parameter estimation
pn and not the unknown exact model parameters p^.
Potential constraints on the choice of the designs can be realized by
constraints on the weight w. For example, the number or the cost of
the measurements can be limited by linear constraints on w. These
constraints have to be considered in the above optimization problem
(Eq. ).
The formulation (Eq. ) is useful if additional
experimental designs should be chosen from a finite number of experimental
designs. Otherwise, the optimization problem can be reformulated so that the
additional optimal design variables have to be optimized directly.
Calculation of optimal experimental designs
A straight-forward way to solve the optimization problem
(Eq. ) is to test all possible values of w.
This direct approach is only practical for small m.
For bigger m, the optimization problem (Eq. ) is
solved approximately. For this purpose, it is solved in the continuous rather
than the discrete setting; that is, the constraint w∈{0,1}m is
relaxed to w∈[0,1]m. Accordingly, the problem
arg minw∈[0,1]mϕ(Mn(w,pn)-1)
is solved.
A possible algorithm to solve this continuous optimization problem is the SQP
algorithm which is, for example, described in chapter 18.
After the continuous problem (Eq. ) is
solved, its solution is projected onto integers with heuristics. An easy way
is to round the continuous solution. Another is to sum up all continuous
weights and then to choose as many designs with the highest continuous
weights. Potential constraints on w still have to be considered by solving
the continuous problem and the following projection onto an integer solution.
The second heuristic, for example, preserves constraints on the number of
designs to choose.
Our numerical experiments with the application examples in
Sect. have shown that the solutions of the
continuous problem (Eq. ) are already close
to integer values. This behavior was also observed, for example, in
and .
Robust optimal experimental designs
The information matrix Mn depends on the estimated parameters pn
if the parameters are nonlinear. This may lead to suboptimal designs if
∇pf(⋅,pn) differs strongly from ∇pf(⋅,p^).
For this reason, we now consider a method which takes into account a possible
nonlinearity of the parameters. This robust method was presented in
and .
The main idea of the method is not to optimize the quality of the covariance
matrix for a single parameter vector pn as in
Eq. (), but to optimize the worst case quality
within a whole domain which contains the unknown exact parameter vector
p^ with high probability.
For this purpose, a confidence region which contains p^ with
probability α∈(0,1) is approximated by
Gn(α):=p∈Rnp|‖p-pn‖Mn(pn)-12≤γ(α).
Here, γ(α) is the α quantile of the χ2 distribution
and ‖v‖A:=vTAv denotes
the energy norm of the vector v∈Rnp with respect to
the positive definite matrix A∈Rnp×np.
The approximation of the confidence region arises from linearization of the
model function f in point pn and the assumption Pn∼N(p^,Mn(pn)-1).
If the worst case quality in the entire region Gn(α) shall be
optimized, the optimization problem (Eq. ) becomes
arg minw∈{0,1}mmaxp∈Gn(α)ϕ(Mn(w,p)-1).
This minimum–maximum optimization problem can be solved only with
considerably more computational effort compared to the optimization problem
(Eq. ). In order to reduce this effort, the
function ϕ(Mn(w,⋅)-1) is linearized in point pn
in the following way:
ϕ(Mn(w,p)-1)≈ϕ(Mn(w,pn)-1)+∇p(ϕ(Mn(w,p)-1))T(p-pn).
The resulting inner maximization problem can be solved analytically. It is
maxp∈Gn(α)ϕ(Mn(w,pn)-1)+∇p(ϕ(Mn(w,p)-1))T(p-pn)=ϕ(Mn(w,pn)-1)+γ(α)12‖∇p(ϕ(Mn(w,pn)-1))‖Mn(pn),
as can be seen, for example, in . With this approach the
optimization problem (Eq. ) is replaced by
arg minw∈{0,1}mϕ(Mn(w,pn)-1)+γ(α)12‖∇p(ϕ(Mn(w,pn)-1))‖Mn(pn).
This optimization problem again can be solved approximatively by solving the
corresponding continuous problem and projecting this solution onto an integer
solution as described in the previous subsection.
It should be noted that in this approach
(Eq. ), the first and second derivatives
of the model are used. In contrast, only the first derivative is used for
local optimal designs (Eq. ).
Efficiency of experimental designs
A common way to describe the benefit of an experimental design is its
efficiency. The efficiency of an experimental design w∈{0,1}m
regarding a criterion ϕ and with n previous measurements is defined as
follows:
Eϕ(w):=minw^∈{0,1}mϕ(Mn(w^,p^)-1)ϕ(Mn(w,p^)-1).
It should be noted that the searched parameter vector p^ is used
here. If this is not known then the efficiency can not be calculated.
The efficiency is always between 0 and 1 and is larger the better the
experimental design is.
The Optimal Experimental Design Toolbox
We implemented the methods presented in the previous section for optimization
of model parameters and experimental designs as a MATLAB toolbox named the
Optimal Experimental Design Toolbox.
MATLAB () was chosen because it supports vector and matrix
operations and provides many numerical algorithms, especially for
optimization. Moreover, MATLAB supports object oriented programming and
therefore permits a simple structuring, modification and extension of the
implementation. Another advantage of MATLAB is that it can easily interact
with C and Fortran.
The toolbox is available at a Git repository ()
under the GNU General Public License (). It includes extensive
commented source code, a detailed help integrated in MATLAB and a user
manual.
Provision of the model function
For the methods described in Sect. ,
the model function and its first and second derivative with respect to the
model parameters are required.
Actually, the model function is required for the parameter optimization and,
depending on the optimization method, also its first derivative. Its first
derivative is also required for the experimental design optimization. If the
robust method is used its second derivative is also required.
The model interface prescribes how to provide these functions. They
need not be written in MATLAB itself, since MATLAB can call functions in C,
C++ or Fortran.
The toolbox has several possibilities to provide the derivatives
automatically. The model_fd class, for example, provides
the derivatives by approximation with finite differences. If the model
function is given as an explicit symbolic function, the
model_explicit class can provide the derivatives by
symbolic differentiation with the Symbolic Math Toolbox.
Listing shows, for example, how a
model_explicit object is created.
Create a model with a symbolic model
function.
![]()
In the event that the model function is given as a solution of an initial
value problem, the Optimal Experimental Design Toolbox contains the
model_ivp class. This class solves the parameter dependent
initial value problem and calculates the necessary derivatives.
Listing shows how a model_ivp object is
created.
Create a model with a model function
given as solution of an initial value problem.
![]()
The class takes advantage of the fact that the integration and
differentiation of the differential equation can be interchanged if the model
function is sufficiently often continuously differentiable. Required
derivatives of the differential equation and initial value are calculated
again by symbolic differentiation with the Symbolic Math Toolbox.
The resulting initial value problems are solved with MATLAB's ode23s
function which can also solve stiff problems. Since the arising initial value
problems for the derivatives are mutually independent, the solutions of the
initial value problems can be calculated in parallel using the
Parallel Computing Toolbox.
Setup of the solver
Methods for the optimization of model parameters and experimental designs are
provided by the solver class. First, a solver object has to
be created and the necessary information has to be passed.
On the one hand, this is the model which has to be set by the
set_model method (see Listing ).
Set the model.
![]()
On the other hand, an initial guess of the model parameters have to be set by
the set_initial_parameter_estimation method (see
Listing ).
Set the initial parameter
estimation.
![]()
Potential accomplished measurements can be set via the
set_accomplished_measurements method. These measurements
consist of the corresponding experimental designs together with their
variances of the measurement errors. Furthermore, the measurement results themselves
have to be passed for a parameter estimation (see
Listing ).
Set accomplished
measurements.
![]()
Finally, if an optimization of experimental designs shall be performed, the
selectable measurements have to be set by the
set_selectable_measurements method (see
Listing ). These measurements consist of
the experimental designs as well as the variances of the measurement errors.
Set selectable
measurements.
![]()
Optimization of experimental designs and<?xmltex \hack{\break}?> model parameters
Once the solver object is configured as described in the previous
subsection, experimental designs or model parameters can be optimized via the
get_optimal_measurements (see
Listing ) or the
get_optimal_parameters (see
Listing ) method, respectively. Constraints on
the experimental designs or model parameters can be passed to the
corresponding method.
The get_optimal_measurements method can solve
the optimization problem directly by trying all possible combinations or
approximatively.
For the approximative solving, the continuous problem is solved with the SQP
algorithm (see , Chapter 18) provided by the
fmincon function of the Optimization Toolbox. Its solution
is projected onto an integer solution by the second heuristic described in
Sect. .
The first derivative of the objective function is provided in analytical
form. This saves much of the computing time compared to derivatives
calculated by finite differences. The Hessian matrix is approximated by the
Broyden–Fletcher–Goldfarb–Shanno (BFGS) update
().
MATLAB's SQP algorithm can recover from infinity. If an infinite function
value is reached during the optimization, the algorithm attempts to take a
smaller step. Thus, if the optimization is started with a regular design,
singular designs do not make any trouble.
Optimize experimental
designs.
![]()
The get_optimal_parameters method uses the
trust-region-reflective () or the
Levenberg–Marquard algorithm ()
provided by the lsqnonlin function of the Optimization Toolbox to solve the least squares problem resulting from the parameter
estimation. The first derivative of the objective function is also provided
analytically.
Optimize model
parameters.
![]()
Furthermore, the expected quality of the resulting parameter estimation for
any selection of experimental designs can be calculated using the
get_quality method of the solver object. Thus,
for example, the increase in quality by adding or removing experimental
designs can be determined.
Execution time and memory consumption
The total time required for the optimization of the model parameters or an
experimental design depends crucially on the time required for evaluating the
model function and its first and second derivative with respect to the model
parameters.
When optimizing model parameters, the model function and its first derivative
has to be evaluated several times with different model parameter vectors at
the accomplished measuring points. When optimizing experimental designs, the
model function and its first and second derivative has to be evaluated for
one model parameter vector but at the accomplished and selectable measuring
points.
Generally, the execution time increases with the number of parameters, the
number of selectable measurements and the number of accomplished
measurements.
The implementation of this toolbox favors a low execution time of a low
memory consumption. For this reason, (intermediate) results within a method
call and between successive method calls are saved and reused. An example is
multiple occurring matrix multiplications within a method call. Another
example is a re-optimization of designs with other constraints, such as
another maximum number of allowed measurements. Here, the derivatives of the
model function calculated in the previous optimization are reused.
Due to the described caching strategy, the total memory consumption depends
linearly on the number of (accomplished and selectable) measurements and
quadratically on the number of parameters. Nevertheless, it should be
possible to solve problems with hundreds of parameters and thousands of
measurements on a standard computer.
Changeable options
Many settings for the optimization of experimental designs or model
parameters are changeable. These can be altered by the
set_option method of the solver object (see
Listing ). The desired options can be set using
property-value pairs, as already known from MATLAB.
Change an
option.
![]()
Estimation method:
The estimation method for the quality of
experimental designs can be selected by the
estimation_method option. The standard point
estimation method and the robust region estimation method, both
presented in Sect. , are supported.
The region estimation method is the default setting.
Confidence level:
The level of confidence for the confidence region at
the region estimation method, represented by α in
Sect. , can be set by the alpha option.
The default value is 0.95.
Prior parameter estimation:
It can be chosen whether a parameter
optimization should be performed before optimizing experimental designs. This
can be set by the parameter_estimation option and the
values yes or no. To save computational time no previous
parameter optimization is performed by default.
Quality criterion:
The quality criterion, which is applied to the
covariance matrix and represented in Sect. as
ϕ, can also be chosen with the criterion option. The
criterion interface prescribes the syntax of the criterion function
and its necessary derivatives. The trace of the covariance is the default
criterion and implemented by the criterion_A class.
Parameter scaling:
It can be chosen whether model parameter should be
scaled before optimizing experimental designs or the model parameters
themselves. Scaling means a uniform impact of all model parameters and is
enabled by default. The options are
edo_scale_parameters and
po_scale_parameters with the values yes
and no.
Optimization algorithm for experimental design:
The exact and the
approximative approach for the optimization of an experimental design problem
can be chosen with the edo_algorithm option and the values
direct and local_sqp. For time reasons, by
default the experimental design problem is solved by the approximative
approach. Furthermore, the number of function evaluations and iterations by
the SQP algorithm can be constrained by the options
edo_max_fun_evals and
edo_max_iter.
Optimization algorithm for parameter estimation:
The optimization
algorithm for the parameter estimation problem can be chosen with the
po_algorithm option. The trust-region-reflective
() and the Levenberg–Marquard algorithm
() can be chosen with the values
trust-region-reflective and Levenberg–Marquardt The
trust-region-reflective algorithm is the default algorithm. Furthermore, the
number of function evaluations and iterations can be limited through the
options po_max_fun_evals and
po_max_iter.
Help and documentation
The Optimal Experimental Design Toolbox also provides extensive
integrated help. Besides system requirements and version information, a
user's guide with step-by-step instructions on how to optimize experimental
designs and model parameters is included. Demos show how to work with the
toolbox in practice. In addition, a detailed description for every class and
method is available.
The layout of the help for the Optimal Experimental Design Toolbox
is based on the design of the help also used by MATLAB and other toolboxes.
Thus, the user does not have to get reoriented with a new layout.
Application examples
In this section, numerical experiments together with their results regarding
the optimization of model parameters and experimental designs are presented
for two models of different complexity. Both models describe the sediment
concentration in seawater during tidal inundation of coastal salt marshes.
Coastal salt marshes have an important ecological function with their diverse
flora and as a nursery for migratory birds. Furthermore, they have the role
of dissipating current and wave energy and therefore reducing erosional
forces at dikes and coastal areas.
With these models, the vertical accretion of coastal salt marshes can be
predicted. When considering expected global sea level rise
(), the future ability of coastal salt marshes to adapt to
rising sea levels and thus to survive can be estimated. Depending on these estimates,
measures to protect these salt marshes can be taken.
Calibration of the model parameters requires measurements of suspended
sediment concentration during tidal inundation, which are time-consuming and
laborious. For this reason, it is advantageous to know under which conditions
and how many of these measurements should be carried out.
The models
Both models are zero-dimensional point models, which describe the sediment
concentration in seawater during tidal inundation of coastal salt marshes.
The first model (C2-model) has two model parameters, was described in
and was adapted for a salt marsh in
the Wadden Sea (southeastern North Sea), located near Hoernum in the southern
part of the island of Sylt (Germany), by
. The second model (C3-model)
has three model parameters, is an extension of the first model and subject of
current research.
The C<inline-formula><mml:math display="inline"><mml:msub><mml:mi/><mml:mn mathvariant="normal">2</mml:mn></mml:msub></mml:math></inline-formula>-model
The first model is called the C2-model. Here, the sediment concentration
in kg m-3 is modeled by the function C:[tS,tE)→R+. Furthermore, tS is the start time of the
inundation of the salt marsh and tE the end time. The
concentration C is given implicitly as the solution of the initial value
problem
C′(t)=-wSC(t)+(C0-C(t))h′(t)h(t)-E if h′(t)>0-wSC(t)h(t)-E else for all t∈(tS,tE) and C(tS)=C0.
Here, C0≥0 is the initial sediment concentration of the flooding
seawater and wS≥0 the settling velocity of the suspended
sediment in m s-1. Moreover, the function
h:R→R,t↦a1+t-x0b2+hHW-hMHW
describes the time-dependent water surface elevation and E the elevation of
the marsh both in meters and relative to a fixed datum. Here, a, b and
x0 are constants describing the change in the water level,
hMHW the mean high water level and hHW the high water
level of a certain tidal inundation in meters. The start tS and end time
tE of the inundation are the points where the
height h equals the elevation of the marsh E.
The sediment concentration C thus decreases continuously within a tidal
cycle depending on the settling velocity wS which is described by
the term
-wSC(t)h(t)-E
in Eq. (). During the flood phase, the reduced sediment
concentration is partially compensated by new inflowing sea water. This is
described by the term
(C0-C(t))h′(t)h(t)-E
in the first case of Eq. ().
The values used in the water surface elevation function h, for the local
salt marsh, are shown in Table . These have been
estimated by nonlinear regression analysis using local historic tide gauge
data from 1999 to 2009 (at Hoernum Hafen, Germany). The continuous
high-resolution (6 min) time series has, therefore, been split into the
individual tidal cycles beforehand
().
Values used for the water surface elevation function h
a
b
x0
hMHW
E
local value
3.7506
19447.1
-1301.0
3.75 m
1.3m
The high water level hHW of the current tidal inundation is
measured or taken from predictions.
The initial sediment concentration C0 and the settling velocity
wS are only roughly known and therefore model parameters.
Reference values derived from literature values and typical ranges can be
found in Table (see , for
C0 and , for C0 and
wS).
Values for the C2-model.
C0 [kg m-3]
wS [m s-1]
reference value
0.1
10-5
typical range
0.01–0.2
4×10-6–4×10-4
start value
5
2×10-7
optimization bound
10-4–104
10-8–1
The C<inline-formula><mml:math display="inline"><mml:msub><mml:mi/><mml:mn mathvariant="normal">3</mml:mn></mml:msub></mml:math></inline-formula>-model
The second model is an extension of the C2-model and is called the
C3-model. Here the model parameters C0 and wS are
substituted by
C0=k(hHW-E),wS=r(C0)s=rks(hHW-E)s.
Where k≥0, r≥0 and s≥0 are unknown model parameters.
Reference values derived from literature values and typical ranges (where
available) can be found in Table (see
, and , for the settling index
s and , for k).
In this model, a linear relationship between the initial sediment
concentration and the high water level is assumed, where during heavy
flooding a higher sediment concentration is assumed
().
Additionally, a relationship between the initial sediment concentration and
the settling velocity is assumed (). This is an empirical
approximation of the so-called flocculation process ().
Values for the C3-model.
k
r
s
reference value
0.25
10-5
0.5
typical range
0.04–0.2
0.5–3.5
start value
12.5
2×10-7
3
optimization bound
10-4–104
10-8–1
10-1–5
Numerical experiments
We performed several numerical experiments to compare the benefit of
optimized with unoptimized measurement conditions. Also, the benefit of
different approaches to optimization measurement conditions was compared.
Using these results, an appropriate approach for the optimization of
conditions for real measurements was selected.
The approaches introduced in Sect.
and implemented by the Optimal Experimental Design Toolbox described
in Sect. were used for the numerical experiments. For
that, we used the model_ivp class which allows for the
calculation of the solution of an initial value problem and its first and
second derivatives with respect to the model parameters. The C2-model was
implemented by the model_C2 class and the C3-model by
the model_C3 class which is a subclass of the
model_C2 class.
For our numerical experiments, we used the model output with the reference
parameters in Tables and plus an
additive normally distributed measurement error with zero expectation as
artificial measurement results. As standard deviation of the measurement
error, we chose 10-2 once and 10-1 once.
In our numerical experiments, we alternately selected a fixed number of
experimental designs and estimated the model parameters with corresponding
measurement results. We carried out each experiment 10 times and averaged
the results to minimize the influence of randomness.
For the parameter estimation, the start values and bounds in
Tables and were used. The bounds
were chosen so that the typical range of values is covered, but also more
extreme values are possible. The starting values were chosen slightly outside
the typical ranges to represent a poor initial guess.
The experimental designs for these models consist of the time point of the
measurement and the high water level of the tidal inundation. A set of thirty
selectable experimental designs was specified. They were obtained by
combining three different high water levels of the tidal inundation (1.5, 2.0
and 2.5 m) with 10 time points equidistantly spread over the inundation
period.
For choosing the experimental designs, we compared the standard and the
robust approach presented in Sect. with the trace as
quality criterion together with uniformly distributed experimental designs.
In the robust approach, a confidence level of 95 % was used. The
optimization problems for the experimental designs were once solved exactly
and once approximatively (see Sect. ). To evaluate
all these methods, we compared the resulting parameter estimations with the
reference model parameters.
We further investigated whether the number of measurements after which new
experimental designs are optimized had an impact on the accuracy of the
parameter estimation. For this purpose, different numerical experiments were
performed where the parameters and experimental designs have been optimized
after each one, three and five measurements. Altogether 50 measurements
were simulated at each experiment with the C2-model. For the C3-model,
150 measurements were simulated at each experiment since the
model is more complex and therefore a sufficiently accurate estimation of its
parameters might be more difficult.
Accuracy of the parameter estimations
In this subsection, we compare the accuracy of the parameter estimations
resulting from the previously described numerical experiments. Some results
are illustrated in Figs. and
.
Results for the C<inline-formula><mml:math display="inline"><mml:msub><mml:mi/><mml:mn mathvariant="normal">2</mml:mn></mml:msub></mml:math></inline-formula>-modelAveraged error in the parameter estimation from 10 optimization
runs with the C2-model and three measurement per iteration with standard
deviation 10-2 of the measurement error.
![]()
Averaged error in the parameter estimation from 10 optimization
runs with the C3-model and three measurement per iteration with standard
deviation 10-2 of the measurement error.
![]()
The accuracy of the parameter estimations for the C2-model only improved
marginally after four to twelve measurements independent of the choice of
the experimental designs. The accuracy improved faster the more frequently
the experimental designs and parameters were optimized. However, the best
achieved accuracy was independent of the frequency.
With uniformly distributed experimental designs the best achieved accuracy
was slightly worse than with optimized experimental designs. Four to six more
measurements were needed compared to optimized experimental designs in order
to achieve their accuracy.
Although the parameters occur nonlinearly in this model, it made close to no
difference whether the standard or the robust approach for the optimization
of the experimental designs was used.
The approximate solving of the discrete optimization problem has resulted in
slightly worse accuracy at the first iterations compared to the exact
solving. Thereafter, the difference was very small. The solutions of the
relaxed continuous optimization problems were almost always nearly integer.
The different standard deviations of the measurement errors only influenced
the best achieved accuracy which was of course worse at a higher standard
deviation. This can be explained by the fact that different constant standard
deviations only mean a different scaling of the objective of the experimental
design optimization problem. Thus, different constant standard deviations do
not affect its solution.
Results for the C<inline-formula><mml:math display="inline"><mml:msub><mml:mi/><mml:mn mathvariant="normal">3</mml:mn></mml:msub></mml:math></inline-formula>-model
After 10–25 measurements, the accuracy of the parameter
estimations for the C3-model with optimized experimental designs only
improved slightly. Again, the fewer
measurements performed per iteration the faster the accuracy improved, and the best achieved accuracy was
independent of the number of measurements per iteration.
With uniformly distributed experimental designs, the best accuracy was
achieved after 24–60 measurements. Furthermore, the best
achieved accuracy was worse by about a factor of 10 compared to the best
accuracy achieved by (standard) optimized experimental designs.
The standard approach for optimizing experimental designs resulted in a
slightly better accuracy compared to the robust approach.
For both approaches, the difference between the accuracy achieved with the
exact solutions of the discrete optimization problem and the accuracy
achieved with the approximate solutions was small but recognizable and almost
constant over the iterations. Also in these experiments, the solutions of the
relaxed continuous optimization problems were almost all nearly integer.
Again, the different standard deviations of the measurement errors only
influenced the best achieved accuracy.
Conclusions regarding the approach for optimizing experimental designs
Optimized experimental designs provided a much more accurate parameter
estimation than uniformly distributed experimental designs independent of the
chosen optimization approach. Furthermore, only about half as many
measurements were needed to archive the same accuracy with optimized
experimental designs as with uniformly distributed experimental designs. In
the more complex model, the difference was even bigger.
The robust approach did not achieved higher accuracy compared to the standard
approach. In the complex model, the robust approach was even slightly less
accurate. This may indicate that the gain in accuracy by taking into account
the nonlinearity is offset by the additional approximations in the robust
approach. Since a considerably higher computational effort is associated with
the robust approach, the standard approach should be preferred, at least for
these models.
The exact solutions of the discrete optimization problems yielded only
slightly better accuracy gains compared to its approximate solutions. The
fact that the approximate solutions were almost all nearly integer also
argues for the approximate solving. This circumstance was also observed in
and . For these
reasons and because the exact solving requires much more computational
effort, the approximate solving should be preferred, at least for these
models.
Efficiency for the experimental designs
We also calculated the efficiencies of the used experimental designs. Some
results are illustrated in Figs. and
.
Averaged efficiency for the experimental designs from 10
optimization runs with the C2-model and three measurement per iteration
with standard deviation 10-2 of the measurement error.
![]()
Averaged efficiency for the experimental designs from 10
optimization runs with the C3-model and three measurement per iteration
with standard deviation 10-2 of the measurement error.
![]()
The results emphasized the already seen importance of the optimization of the
experimental designs. In particular, the advantage in the case of the few
measurements carried out so far was highlighted. Again, the slight advantage
of the standard approach over the robust approach was visible. With
increasing number of accomplished measurements, the selection strategy of new
measurements became less important as the amount and thus the influence of
the new measurements compared to those of the accomplished measurements
decreased.
Distribution of optimal measuring points
In this subsection, we compare the distribution of the measuring points
optimized in the previously described numerical experiments. Graphical
representation of the distribution of the measuring points from some
numerical experiments are shown in
Figs. and
.
Distribution for the C<inline-formula><mml:math display="inline"><mml:msub><mml:mi/><mml:mn mathvariant="normal">2</mml:mn></mml:msub></mml:math></inline-formula>-model
Averaged frequency of measurements from 10 optimization runs with
the C2-model and three measurement per iteration with standard deviation
10-2 of the measurement error.
![]()
The optimized measuring points were almost exclusively located at the start
and end of the inundation periods. At the start of the inundation period,
both approaches in the exact variant favored lower high water levels unlike
both approaches in the approximate variant which favored higher high water
levels. At the end of the inundation period, the standard approach in both
variants favored lower high water levels unlike the robust approach in both
variants which favored higher high water levels.
Distribution for the C<inline-formula><mml:math display="inline"><mml:msub><mml:mi/><mml:mn mathvariant="normal">3</mml:mn></mml:msub></mml:math></inline-formula>-model
Averaged frequency of measurements from 10 optimization runs with
the C3-model and three measurement per iteration with standard deviation
10-2 of the measurement error.
![]()
For the C3-model the optimized measuring points accumulated at the end of
the inundation periods. All approaches favored lower high water levels. With
an increasing number of measurements per iteration, the robust approach in
both variants also preferred measurements in the middle of the inundation
periods with the highest high water level.
Conclusions regarding the distribution of optimal measuring points
The numerical experiments showed that measurements at the start and end of
the inundation periods should be preferred for the C2-model.
Measurements at the start of the inundations can be justified by the fact
that one parameter of the model is the concentration at the start of the
inundation. The fact that the settling velocity as second model parameter
most affects the concentration at the end of the inundations justifies
measurements here. This can be confirmed by an examination of the ordinary
differential equation of the model derived with respect to the settling
velocity. The derivative of the model with respect to the settling velocity
is zero at the start of the inundation and is getting smaller the further the
inundation progresses. Its absolute greatest value it thus reached at the end
of the inundation.
The experiments with the C3-model showed that here measurements at end of
the inundation periods should be preferred. In this model, the concentration
at the start is no parameter but is affected by a parameter that also
influences the settling velocity. For this reason, measurements are not
suggested at the start.
For both models the high water level seemed to play a minor role for the
choice of measuring points.
As a rule of thumb, one can say that measurements should be carried out at the
end of an inundation period and also some at the start if the C2-model is
used.
Conclusions
In this paper we presented two different approaches for optimizing
experimental design for parameter estimations. One method was based on the
linearization of the model with respect to its parameters, the other takes
into account a possible nonlinearity of the model parameters. Both methods
were implemented in our presented Optimal Experimental Design Toolbox for MATLAB.
By employing the presented approach for two existing salt marsh models, we
showed that model parameters can be determined much more accurately if the
corresponding measurement conditions were optimized. In particular for
time-consuming or costly measurements, it is useful to optimize the
measurement conditions with the Optimal Experimental Design Toolbox.
This gain in accuracy is not limited to the application examples. In general,
using the implemented methods, the accuracy of the parameters of any model
can be maximized while minimizing the measurement cost, provided that the
related assumptions are fulfilled. However, the required execution time for
the optimization increases with the number of model parameters and
(accomplished and selectable) measurements. Parallelization techniques in the
optimization as well as in the model evaluation itself can counteract this
effect.
In addition to the parallelization, the optimization in the toolbox could
also be extended to techniques of globalization, so that the chance of
getting into a local minimum is reduced.
The results concerning the application examples have not significantly
differed despite the various approaches for optimizing experimental design.
For this reason, the approach with the least computational effort is
recommended. However, this recommendation can not be applied readily to other
(more complex) models. Here, the performance of the approaches should be
compared again if possible.
Furthermore, it has been found that measurements at the beginning and end of
the inundation period are particularly important for the application
examples. The high water level of the inundation seemed to play a minor role.
These results, however, can not be applied easily to other models. Usually, a
separate optimization of experimental design makes sense here.
Code availability
The Optimal Experimental Design Toolbox is available under the GNU
General Public License () at a Git repository
(). In addition to the toolbox, including commented
source code and a user manual, an implementation of the application examples
is also available.