The CLIMADA engine integrates the input variables exposure (E), hazard (H), and impact function (F) for risk assessment. For the appraisal of adaptation options, the exposure and impact function are combined with the adaptation measure (M) in a container input variable called entity (T). Note that further input variables might be added in future versions of CLIMADA. Each of these input variables comes with any number of uncertainty input parameters α, distributed according to an independent probability distribution pα. An input variable can have any number of uncertainty input parameters, and there is no restriction on the type of probability distributions (uniform, Gaussian, skewed, heavy-tailed, discrete, etc.). In the current implementation, any distribution from the Scipy.stats Python module is accepted. The input parameters can define any variation or perturbation of the input variables (e.g. initial conditions, boundary conditions, forcing inputs, resolutions, normative choices, etc.).

In literature, the terminology “input factor” instead of “input parameter” is also used. Here we shall exclusively use the terminology “input parameter” for numerical random variables, and “input variable” for the inputs to the CLIMADA model.

In general, there are two basic approaches regarding how samples can be drawn. In the local “one-at-a-time” approach, the input parameters are varied one after another, keeping all the others constant . Local methods are conceptually simpler, but capture neither interactions between input parameters nor non-linearities . By contrast, in global methods, the input parameters are sampled from the full space at once . This allows for a more comprehensive depiction of model uncertainty by accounting for the interactions among the input parameters. even argue that uncertainty and sensitivity analyses should always be based on global methods for models with non-linearities such as CLIMADA.

Hence, the basic premise of the unsequa module is to use a global sampling algorithm based on (quasi-) Monte Carlo sequences to generate a set of N samples of the input parameters. Here, one sample refers to one value for each of the input parameters. Following the heat wave example described in Sect. , one would create N global samples xn=(tn,sn,an) with n∈[1,…,N]. One sample thus corresponds to a set of three numbers in this case. Choosing the correct number of samples is a notoriously difficult task . One generic approach is to start with a sample size that one can afford to generate reasonably efficiently (e.g. N∼100D), and then check the confidence intervals of the estimated sensitivity indices (cf. Sect. ). If relative values of the estimated indices are too ambiguous to draw key conclusions due to the overlap of confidence intervals, one should either generate more samples, or use a more frugal method (e.g. reduce the number of input parameters D) .

CLIMADA imports the (quasi-) Monte Carlo sampling algorithms from the SALib Python package . Thus, all sampling algorithms from this package are directly available to the user within the unsequa module. These algorithms are all implemented for a uniform distribution over [0,1] at least. In order to accommodate any input parameter distributions, the unsequa module uses the percent-point function (ppf) of the target probability density distribution (cf. Appendix ).

For each sample of the input parameters, the model output metrics are computed using the CLIMADA engine, e.g. for the risk assessment, the impact matrix In for each sample xn. Following the heat wave example from the previous section, for each sample xn=(tn,sn,an) of the input parameters, the algorithm first sets the input variables E(n)=E(tn),H(n)=H, and F(n)=F(sn,an). Second, the corresponding impact matrix In is computed for each sample independently, following the algorithm described in . All CLIMADA risk output metrics such as the average annual impact, the exceedance frequency curve, or the largest event are then derived from the matrix In and the hazard frequency defined in H(n).

Similarly, for the appraisal adaptation options, each sample is assigned with the corresponding input variables. The CLIMADA engine is then used to compute the impact matrix Inm(y) for each sample n, each adaptation measure m, and each year y following the algorithm described in . All CLIMADA benefit and cost metrics such as the total future risk, the adaptation-measure benefits, the risk transfer options, and costs are derived from the impact matrix Inm(y), the adaptation measure Mm(n,y), the exposure E(n,y) and the hazard H(n,y). Note that in practice, the input variables for the exposure, impact function, and adaptation measure are combined into one input variable called entity T(n,y), which also includes information about optional discount rates and risk transfer options.

We remark that no direct evaluation of the convergence of this quasi-Monte Carlo scheme is provided in the unsequa module, as it is not generally available for all the possible sampling algorithms available through the SALib package. Instead, the sensitivity analysis algorithms, to be described in Sect.  below, provide confidence intervals. In SALib, confidence intervals relate to the bounds which cover 95 % of the possible sensitivity index value, estimated through bootstrap resampling. These can be used as a proxy to assess the convergence of the uncertainty analysis. If the intervals are large and overlapping, the result is likely not robust and the number of samples should be increased.

In all of the uncertainty and sensitivity analyses, computing the model outputs is usually the most expensive step computationally. For convenience, an estimation of the total computation time for a given run is thus provided in the unsequa module. Experiments showed that the computation time scales approximately linearly with the number of samples N; it is also proportional to the time for a single impact computation. The latter is mostly defined by the size of the exposure (i.e. depends on the resolution, size of the considered geographical area, etc.) and the size of the hazard (i.e. depends on the number of events, the centroid's resolution, etc.). In case the input variables are generated using an external model (e.g. a hydrological flood model for the hazard), the computation time is also proportional to the external model run time. For complex models, this can be prohibitively long. In such cases, one can pre-compute the samples for the given input variable, thus trading CPU time for memory (cf. Litpop example in Sect. , and the helper methods in Appendix ). The number of samples N in turn scales with the dimension D (i.e. the number of input parameters), depending on the chosen sampling method. For the default unsequa module, Sobol′ method, the scaling is O(D). In addition, for the appraisal of adaption options, the risk computation is repeated for each of the Nm adaptation measures. This results in a total computation-time scaling of O(D) for the risk assessment, and O(D⋅Nm) for the appraisal of adaptation options. Thus, for large number of input parameters, and/or long single impact computation times, and/or large numbers of adaptation measures, the computation time might become intractable. In this case, one could consider using surrogate models , a feature that might be added to future iterations of the unsequa module.

The output metrics values for each sample are characterized and visualized. To this effect, various plotting methods have been implemented as shown in Sect.  and . For instance, it is possible to visualize the full distributions or compute any statistical value for each model output metric. The key objective is to obtain an understanding of the uncertainties in the model outputs beyond the mean value and standard deviation.

The sensitivity index Sα(o) is a number that subsumes the sensitivity of a model output metric o to the uncertainty of input parameter α . Since CLIMADA is a non-linear model, only global sensitivity indices are suitable . To derive such global sensitivity indices, several algorithms are made available through the SALib Python package , including variance-based (ANOVA) , elementary effects , derivative-based , FAST , and more . Importantly, each method requires a specific sampling sequence to compute the model output distribution and results in distinct sensitivity indices. These distinct indices will typically agree on the general findings (e.g. what input parameter has the largest sensitivity), but might differ in the details as they correspond to fundamentally different quantities (e.g. derivatives against variances). The recommended pairing of the sampling sequence and sensitivity index method is described in the SALib documentation, and simple save-guard checks have been implemented in the unsequa module. Note that it is technically valid to use different sampling algorithms for the uncertainty and for the sensitivity analyses. For example, one can first use sampling algorithm A to perform an uncertainty analysis, i.e. steps from Sect. –. Then, one can use another sampling algorithm B as required for the chosen sensitivity index algorithm to perform the sensitivity analysis, i.e. steps from Sect. – and , . However, in practice, since generating samples is often the computational-time bottleneck, it is more convenient to use the same methods so that the same samples can be used for both analyses .

For typical case studies using CLIMADA, Sobol′ indices are generally well-suited for both uncertainty and sensitivity analyses. For sampling the algorithm, the use of the Sobol′ quasi-Monte Carlo sequence is required, which provides good rates of convergence when the number of input parameters is lower than ∼25 . Sobol′ indices are obtained as the ratio of the marginal variances to the total variance of the output metric. In particular, the algorithm implemented in the SALib package allows us to estimate the first-order, total-order, and second-order indices . First-order indices measure the direct contribution to the output variance from individual input parameters. Total-order indices measure the overall contribution from an input parameter considering its direct effect and its interactions with all the other input parameters. Second-order indices describe the sensitivity from all pairs of input parameters. In addition, the 95th percentile confidence interval is provided for all indices. This allows us to estimate whether the number N of chosen samples was sufficient for both the uncertainty and sensitivity analysis. Note that in general, the rate of convergence depends non-trivially on the number of input parameters, the probability distributions of the input parameters, the type of sensitivity index, and the sampling algorithm .

The last step consists of analysing and visualizing the obtained sensitivity indices. To this effect, a series of visualization plots are provided, such as bar plots or sensitivity maps for first-order indices, and correlation matrices for second-order indices, as shown in Sect.  and . This step shows which input parameters' uncertainty is the driver of the uncertainty of each individual module output metric. This is useful to support model calibration and verification, to prioritize efforts for uncertainty reduction, and to inform robust decision-making.

We identified four main quantifiable uncertainty parameters which are summarized in the upper row of Table . As we remarked above, the choice of the distribution of input parameters can substantially influence the results of the uncertainty and sensitivity analyses; it should thus ideally be based on background knowledge. The distributions chosen here are plausible, yet stylized, and should not be considered as general references for other case studies.

For the exposure, the total population is assumed to be subject to random sampling errors that are well captured by a normal distribution, and a maximum error of ±10 % is assumed. Thus, the total population is scaled by a multiplicative input parameter T, distributed as a truncated Gaussian distribution, with clipping values 0.9,1.1, mean value μ=1, and variance σ=0.05. For the population distribution, the original case study used the Gridded Population of the World (GPW) dataset , which is available down to admin-3 levels. To account for uncertainties arising from the finite resolution, we use the CLIMADA's LitPop module to enhance the data with nightlight satellite imagery from the Black Marble annual composite of the VIIRS day–night band (Grayscale) at 15 arcsec resolution from the NASA Earth Observatory , a common technique used to rescale population densities to higher resolutions . In LitPop, the nightlight and population layers are raised to an exponent m and n, respectively, before the disaggregation. Here, we vary the value of m and n as a description of the uncertainty in the population distribution. In the original case study, m is set to 0 and n to 1. We consider the addition of the nightlight layer with m∈(0,0.5,1), and vary the population layer with n∈(0.75,1,1.25). The corresponding distributions are shown in Fig. . A higher value of n emphasizes highly populated areas, a lower value the sparsely populated areas. The corresponding input parameter L represents all pairs of (m,n).

For the hazard, we apply a bootstrapping technique, i.e. uniform resampling of the event set with replacement to account for the uncertainties of sample estimates. Since the default Sobol′ global sampling algorithm requires repeated application of the same value of any given input parameter, here we define H as the parameter that labels a configuration of the resampled events. Errors from the hazard modelling (cf. Appendix ) are not further considered here. A more detailed study might want to explore further uncertainty sources, such as the wind-field model, the hazard resolution, or the random event set generation algorithm.

Finally, for the impact function, we consider the uncertainty in the threshold of the original step function that was used to estimate the number of people “affected” (widely defined) by storm surges. In the original case study, the threshold was 1 m, with 0 % affected people below, and 100 % affected people above. We consider a threshold shift S between 0.5 and 3 m. This extends a range examined in a study of human displacement due to river flooding (there from 0.5–2 m) , in order to extensively explore the uncertainty related to resolution of the population and topography. This distribution does not examine a specific impact, but rather how the total number of people affected varies based on different thresholds used to define “affected”. The resulting range of the impact function is shown in Fig. .

We use the default Sobol′ sampling algorithm to generate a total of 10240 samples as shown in Fig. .

For each of the samples n, the full impact matrix In is obtained and saved for later use. Furthermore, from the impact matrix, we compute several risk metrics for each sample: the average annual impact aggregated over all exposure points, the aggregated risk at returns periods of 5, 10, 20, 50, 100, and 250 years, the impact at each exposure point, as well as the aggregated impact for each event (for details cf. ).

In the following discussion, we concentrate on the analysis of the full uncertainty distribution of various risk metrics. For convenience, the original case study value, the uncertainty mean value, and standard deviation are also reported. However, as we shall see below, focusing only on these numbers would provide a limited picture.

The full uncertainty distribution for each of the return periods, as well as the exceedance frequency curve are shown in Fig. . First, we remark that the exceedance frequency curve of the original case study, shown in Fig. b, is close to the median percentile, while the upper and lower 95th percentiles of the uncertainty are roughly +40 % and -60 % compared to the median, respectively. Second, the distribution of uncertainty for each return period separately, shown in Fig. a, is in fact bimodal, particularly for shorter return periods. The original case study values for the lower return periods are all among the higher mode. Third, the distribution of the average annual impact aggregated over all exposure points, shown in Fig. c, is also bimodal, with the original case study lying in the mode with larger impacts. The mean number of affected people is 1.42 M with a variance of ±1.03 M, which is compatible with, but lower than the original case study value of 1.94 M.

The bimodal form of the impact uncertainty distribution is interesting, as one could rather expect statistical white or coloured noise (e.g. Gaussian or power-law distributions). As a consistency proof that this is not due to a computational setup error, we verified that the distribution of the total asset value, shown in Fig. d, aligns with the parametrization of the exposure uncertainty (cf. Table ). For a better understanding of the obtained uncertainty distributions, particularly understanding the bimodality, let us continue with the sensitivity analysis.

Ideally, we should choose the sensitivity method best suited for the data at hand. In our case, the uncertainty distribution is strongly asymmetric (cf. Fig. ), thus a density-based approach would be best . However, this would require generating a new set of samples, and for the purpose of this demonstration, we used the unsequa default variance-based Sobol′ method. Note that despite the questionable use of variances to characterize sensitivity for multi-modal uncertainty distributions, the derived indices prove useful to better understand the results from the case study at hand.

We thus computed the total-order and the second-order Sobol′ indices for all the input parameters T, L, H, and S. We obtained the sensitivity indices for all the risk metrics shown in Fig. : average annual impact aggregated (aai_agg), impact for return periods of 5, 10, 20, 50, 100, and 250 years (rp5, rp10, rp20, rp50, rp100, rp250), and additionally for the average annual impact at each exposure point.

As shown in Fig. a, for the average annual impact aggregated, the largest total-order sensitivity index is for the impact function threshold shift with STS(aai_agg)≈0.95. This indicates that the uncertainty in the impact function threshold shift S is the main driver of the uncertainty. Thus, to understand the bi-modality of the uncertainty distribution (cf. Fig. c), we have to better understand the relation between S and the model output. Note that there are no strong interactions between the input parameter uncertainties as all second-order sensitivity indices S2≈0 (cf. Fig. ). Thus, it is reasonable to assume that the bi-modality of the distribution comes directly from S and not from correlation with other input parameters. We further remark that the 95th percentile confidence intervals of the sensitivity indices (indicated with vertical black bars in Fig. ) are much smaller than the difference between the sensitivity indices. We thus conclude that the number of samples was sufficient for a reasonable convergence of the uncertainty and sensitivity sampling algorithm.

A further analysis of the average annual impact aggregated value in function of the impact function threshold shift S reveals a discontinuity at a value of Sd∼1.85 m as shown in Fig. a. Hence, the bimodality of the uncertainty distributions (cf. Fig. ) is indeed due to the uncertainty input parameter S of the impact function, but it does not explain the root cause. Further understanding is obtained from studying the storm surge footprint used in the original case study. Plotting the storm surge intensity of all events at each location with values ordered from smallest to largest, we find a discontinuity and plateau around 1.85 m, as shown in Fig. b. This is the value corresponding to the threshold shift at which the average annual impact is discontinuous. Thus, the bimodality of the uncertainty distributions, while caused by uncertainty in the impact function, is rooted in the modelling of the storm surge hazard footprints. Further research beyond the scope of this paper would be needed to understand whether this value of 1.85 m has a physical origin (e.g. landscape features or protection standards), or is due to a modelling artefact. However, despite the discontinuity, the patterns are as expected: an impact function with a step at 0.5 m results in many more people being classified as affected than when the step is at 3 m (in the latter case, only particularly large storm surges would result in people being affected). For planning purposes, the lower end of this impact function shift is most relevant – even 0.5 m depth of a storm surge can be dangerous for people – so the higher mode of the distribution in Fig.  is most relevant.

Finally, the largest sensitivity index for the average annual impact at each exposure point is reported on a map in Fig. b. In the highly populated regions around Ho Chi Minh city (South Vietnam) and Haiphong (North Vietnam), the largest index is S in accordance with the sensitivity of the average annual impact aggregated over all of Vietnam. However, in less densely populated areas, such as the larger Mekong delta (South Vietnam), the outcome is more sensitivity to the population distribution L. Furthermore, while for shorter return periods, the largest total-order sensitivity index is the impact function threshold shift STS, for longer return periods the sensitivity to the population distribution STL becomes larger as shown in Fig. c. This might be because stronger events with large return periods consistently have larger intensities than the maximum threshold shift of 3 m. Together, these results hint to potentially hidden high-impact events in unexpected areas (e.g. a large storm surge in the less densely populated southern tip of Vietnam could affect a large number of people).

We identified four additional quantifiable uncertainty input parameters for the appraisal of adaptation options compared to the risk-assessment study (cf. Sect. ) that are summarized in the bottom row of Table . For the exposure, the growth rate of the population from 2020 to 2050 was estimated at 13 % in the original case study based on data from the United Nations . Here, we assume a growth rate G uniformly sampled between 10 % and 16 %. For the hazard, the original case study used the parameters from to scale the intensity and frequency of the events, considering the climate-change scenario RCP8.5 from 2020 to 2050 (see Appendix  for more details). This method is subject to large uncertainties see e.g. and we thus scale the intensity and frequency with parameters I and F, uniformly sampled from [0.9,1.1] and [0.5,2], respectively. Finally, the cost of the adaptation measures is assumed to vary by a multiplicative factor C, sampled uniformly between [0.5,2].

For the sampling, we use the default Sobol′ sampling algorithm to generate a total of N=18432 samples. Owing to the larger amount of input parameters, the total number of samples is larger than for the risk assessment (cf. Sect. ). The drawn samples are shown in Fig. .

For each of the samples, we obtained the cumulative output metrics over the whole time period 2020–2050. In particular, we obtained the total risk without adaptation measures, the benefits (averted risk) for each adaptation measure, and the cost of each adaptation measure (for details see ). One can then compare the cost–benefit ratios, i.e. the cost in dollars per reduced number of affected people, for each of the adaptation measures including model uncertainties.

The uncertainty for the cumulative, total average annual risk from storm surges aggregated over all exposure points is shown in Fig. d. The distribution is bimodal, which can be traced back to the storm surge model as explained in Sect. . The original case study value is located in the larger mode, similar to the average annual risk in 2020 as discussed in Sect. . This bimodality translates to the uncertainty in the benefit (total averted risk) for the adaptation measure sea dykes, Fig. b, but not to the adaptation measures mangroves and gabions, Fig. a and c. Rather, the latter show a heavy-tail uncertainty distribution. Furthermore, the uncertainty analysis of the ratio of the cost to the benefits for each adaptation measure indicates that, contrary to the original case study, the sea dykes might in fact be the least (instead of the most) cost-efficient adaptation measure (see Fig. a–c). Note that expressing the cost-efficiency of an adaptation measure in terms of a reduced number of affected people for each invested dollar presents ethical challenges as will be discussed in more detail in Sect. .

We use the same method as for the risk assessment to compute the total-order ST and the second-order S2 Sobol′ indices for all the input parameters T,L,G,H,F,I,S,C (cf. Table ). We obtain the sensitivity indices for all the metrics shown in Figs.  and , i.e. the total risk as well as the benefits and cost–benefit ratios for all adaptation measures.

The total risk without adaptation measure is most sensitive to the impact function threshold shift S with STS(total risk)≈0.85 as shown in Fig. b. In addition, the sensitivity to the storm surge frequency changes STF(total risk)≈0.18 is significantly larger than the sensitivity to the intensity changes SI(total risk)≈0.02. This could be a consequence of the choice to use a step function to model the vulnerability.

The uncertainty of the benefits for all adaptation measures are most sensitive to the impact function threshold shift, with STSmangroves(benefit)≈STSgabions(benefit)≈0.85, and STSsea dykes(benefit)≈0.75 as shown in Fig. a. This is consistent with the sensitivity of the risk in 2020 (cf. Fig. ). Furthermore, there is some sensitivity to the people distribution L, and to the uncertainty in the climate-change input parameters I and F. Note, however, that STIsea dykes(benefit)≈0, i.e. the uncertainty of the benefits from the adaptation measure sea dykes is not sensitive to the hazard intensity uncertainty, while it is for both mangroves and gabions. This could be because sea dykes are parameterized to reduce the storm surge level by 2 m, which is above the Sd=1.85 m identified in Sect.  as critical for the surge modelling, while gabions and mangroves are parameterized to provide a reduction of 0.5 m which is below (cf. Appendix  and ). Thus, a change in the hazard frequency and the population distribution patterns will result in a stronger variation of the benefits for sea dykes because fewer, but stronger events contribute to the remaining risk each year.

Note that the 95th percentile confidence intervals of the sensitivity indices (indicated with vertical black bars in Fig. ) are much smaller than the difference between the sensitivity indices. We thus conclude that the number of samples was sufficient for a reasonable convergence of the uncertainty and sensitivity sampling algorithm.