Using an Uncertainty Quantification Framework to Calibrate the Runoff Generation Scheme in E3SM Land Model V1

Xu, Donghui; Bisht, Gautam; Sargsyan, Khachik; Liao, Chang; Leung, L. Ruby

doi:https://doi.org/10.5194/gmd-2021-401

Preprints

https://doi.org/10.5194/gmd-2021-401

Preprints

Submitted as: model evaluation paper

16 Dec 2021

Submitted as: model evaluation paper | 16 Dec 2021

Status: a revised version of this preprint was accepted for the journal GMD and is expected to appear here in due course.

Using an Uncertainty Quantification Framework to Calibrate the Runoff Generation Scheme in E3SM Land Model V1

Donghui Xu¹, Gautam Bisht¹, Khachik Sargsyan², Chang Liao¹, and L. Ruby Leung¹ Donghui Xu et al.

Show author details

¹Atmospheric Sciences and Global Change Division, Pacific Northwest National Laboratory, Richland, WA, USA
²Sandia National Laboratories, Livermore, CA, United States

Received: 29 Nov 2021 – Discussion started: 16 Dec 2021

Abstract. Runoff is a critical component of the terrestrial water cycle and Earth System Models (ESMs) are essential tools to study its spatio-temporal variability. Runoff schemes in ESMs typically include many parameters so model calibration is necessary to improve the accuracy of simulated runoff. However, runoff calibration at global scale is challenging because of the high computational cost and the lack of reliable observational datasets. In this study, we calibrated 11 runoff relevant parameters in the Energy Exascale Earth System Model (E3SM) Land Model (ELM) using an uncertainty quantification framework. First, the Polynomial Chaos Expansion machinery with Bayesian Compressed Sensing is used to construct computationally inexpensive surrogate models for ELM-simulated runoff at 0.5° × 0.5° for 1991–2010. The main methodological advance in this work is the construction of surrogates for the error metric between ELM and the benchmark data, facilitating efficient calibration and avoiding the more conventional, but challenging, construction of high-dimensional surrogates for ELM itself. Second, the Sobol index sensitivity analysis is performed using the surrogate models to identify the most sensitive parameters, and our results show that in most regions ELM-simulated runoff is strongly sensitive to 3 of the 11 uncertain parameters. Third, a Bayesian method is used to infer the optimal values of the most sensitive parameters using an observation-based global runoff dataset as the benchmark. Our results show that model performance is significantly improved with the inferred parameter values. Although the parametric uncertainty of simulated runoff is reduced after the parameter inference, it remains comparable to the multi-model ensemble uncertainty represented by the global hydrological models in ISMIP2a. Additionally, the annual global runoff trend during the simulation period is not well constrained by the inferred parameter values, suggesting the importance of including parametric uncertainty in future runoff projections.

Download & links

Preprint (PDF, 11851 KB)

Supplement (9266 KB)

Donghui Xu et al.

Status: closed

CEC1:
'Comment on gmd-2021-401', Juan Antonio Añel, 03 Jan 2022

Dear authors,
After checking your manuscript, it has come to our attention that it does not comply with our Code and Data Policy.

https://www.geoscientific-model-development.net/policies/code_and_data_policy.html

You must address several issues:

First, you have archived your code in GitHub. However, GitHub is not a suitable repository. GitHub itself instructs authors to use other alternatives for long-term archival and publishing, such as Zenodo. Therefore, please, publish your code in one of the appropriate repositories.

Secondly, you state that the code is 'Matlab code'. This is a wrong statement. The code is 'M' code (the language). Matlab could be the interpreter you have used to run the code, but you should be aware that it could be run with other interpreters, such as the FLOSS alternative GNU Octave. Please, clarify it. If you have used Matlab, make explicit the Matlab version you have used in your work.

Also, we can not accept the ANL systems as a suitable repository to store the ELM input data. You must use one of the repositories mentioned in our policy. For example, FigShare, which you already use for other data, is suitable. The same happens with the ILAMB benchmarks.

Please, reply as soon as possible to this comment with the necessary links and DOIs for the code and data so that they are available for the peer-review process, as they should be. Also, you must include in a potential reviewed version of your manuscript the modified 'Code and Data Availability' section containing links and the DOIs.
Juan A. Añel

Geosc. Mod. Dev. Executive Editor

Citation: https://doi.org/10.5194/gmd-2021-401-CEC1
- AC1: 'Reply on CEC1', Donghui Xu, 03 Jan 2022
  
  Dear Editor,
  Thanks for reminding us regarding the Code and Data Policy, which we overlooked in our initial submission. We uploaded our ELM code, UQTk code, and processing and plotting script on Zenodo. We also clarified the version of Matlab we used to run the processing and plotting scripts. The domain data and surface data that needed for running ELM-v1, and the processed ISIMIP2a runoff benchmark are archived on Zenodo as well. We updated the Code and Data Availability section in the attached manuscript. You can also find the modified section in the following:
  Code and Data Availability
  The current version of ELM is available from E3SM project (https://github.com/E3SM-Project/E3SM/releases/tag/v1.1.0). The UQTk code and documentation are available from https://www.sandia.gov/uqtoolkit/. The exact version of ELM, exact version of UQTk source code, and scripts to produce the plots in this study is archived on Zenodo (https://doi.org/10.5281/zenodo.5815500). Matlab version R2019b Update 4 was used to run the processing and plotting scripts. ILAMB version 2 was used in this study, and the package can be accessed at 10.18139/ILAMB.v002.00/1251621. The domain file and surface data file that used to run ELMv1, and processed ISIMP2a runoff data are archived on Zenodo (https://doi.org/10.5281/zenodo.5815730). The GRUN runoff dataset was downloaded from https://doi.org/10.6084/m9.figshare.9228176.
  Thank you,
  Donghui and coauthors
  
  Citation: https://doi.org/10.5194/gmd-2021-401-AC1
RC1: 'Comment on gmd-2021-401', Anonymous Referee #1, 02 Mar 2022

This work presents a surrogate-assisted framework for calibrating runoff relevant parameters in global-scale Earth System Models (ESMs). The large computation burden arisen from repeated simulations in calibration is alleviated by building fast-to-run PCE-based surrogate models of ESMs. It is concluded that the calibrated model obtains an improved performance compared to the one with default parameter values. In summary, the manuscript is generally well-written and may be eventually accepted after addressing the following comments:

-The title should be revised. In my opinion, uncertain quantification is different from calibration. How can one use a UQ framework to calibrate models? How about “Using a surrogate-assisted Bayesian framework to ...”?

-In table 1, the prior for q_{drai,max} is U(1e-6,1e-1). Why not use a logarithmic transformation for it? Otherwise, much more prior samples will be drawn from, e.g., (1e-2, 1e-1).

-Line 193 and Eq. (17), the authors should clearly present how they determine the values of sigma.

-In section 3.3, which criterion (e.g., the Gelman-Rubin R statistic [Gelman et al., 1995]) is used here to check the convergence of MCMC sampling? From Fig. 6 it can be seen that the posterior ranges are still relatively large, which gives the feeling that the MCMC chain has not totally converged.

-In section 5.2, the poor performance of PCE surrogate models in arid regions probably because of PCE’s inability to approximate highly nonlinear functions (a well-known limitation of PCE) or/and the low signal-to-noise ratio in these regions. The authors should elaborate these to provide more informative results to readers. An alternative surrogate method for approximating highly nonlinear function is the deep neural networks.

-In Figs. 10a and 11, the simulated runoff time series with default parameter values could be even closer to the reference GRUN time series than the calibrated ones, giving the feeling that the calibration is not that satisfactory. Please further explain.

-Line 533, ‘are estimated’, Line 534, ‘are run’?

Citation: https://doi.org/10.5194/gmd-2021-401-RC1
RC2:
'Comment on gmd-2021-401', Anonymous Referee #2, 21 Mar 2022
The manuscript performed a per-grid calibration of the E3SM ELM model against a global monthly runoff dataset. The calibration was enabled by developing surrogate models for each grid of the ELM using Polynomial Chaos Expansion to mimic the response surface, which was chosen to be the root mean square error of monthly runoff for each grid. Subsequent analyses examined the spatial distribution of calibrated parameters with higher sensitivity and parametric uncertainty effects on simulated runoff. The paper is well organized, clearly written, and deals with an important topic of calibrating ELM and similar models. However, I have some concerns regarding the accuracy of surrogate and its effect on calibration, as detailed below.

Line 15: “The main methodological advance in this work is the construction of surrogates for the error metric between the ELM and the benchmark data”. But this is not entirely new as using surrogate in this manner has been done previously, e.g. Wang et al. (2014); Razavi et al. (2012) and references therein.

Line 111: what’s the difference between surface runoff and surface water runoff?

Line 195 - to reduce the log likelihood to least-squares regression, further assumption is needed, which might include constant and known sigma. Please verify.

Line 197 - I am not sure whether 1,000 samples are sufficient for burn-in, since MCMC often requires a large number (e.g., tens of thousands) of samples to converge. Including some convergence check statistics or plots in supplementary material would be helpful. Also, what is the MCMC algorithm being used here? Please include a reference for reproducibility.

Fig. 1& 2 - there’s some discrepancy between RMSE given by the surrogate and by ELM. Studies have shown that even small surrogate error can lead to large deviation of the inferred parameter posterior from the “true” posterior (Laloy and Jacques, 2019). I realize that it is not possible to calibrate ELM at global scale, but it seems possible to perform some quick test to validate the surrogate modeling approach. For example: for a few grids compare the posterior obtained using PCE and using ELM; In Section 3.5, step #4, compare the RMSE of ELM simulation with that of PCE.

Line 355: If I understand correctly, 10,000 is the number of runs of the surrogate. It is not necessarily the case if ELM is run, because the convergence rate may be different given the surrogate error (Razavi et al., 2012).

Fig. 11 - it seems that the same period of 1997-2010 is used to calibrate the model and validate the optimal parameters. Is data available after 2010 for validation, so that validation data is independent from calibration data?

Some paragraphs are indented, some are not.

References

Laloy, E., & Jacques, D. (2019). Emulation of CPU-demanding reactive transport models: a comparison of Gaussian processes, polynomial chaos expansion, and deep neural networks. Computational Geosciences, 23(5), 1193-1215.

Wang, C., Duan, Q., Gong, W., Ye, A., Di, Z., & Miao, C. (2014). An evaluation of adaptive surrogate modeling based optimization with two benchmark problems. Environmental Modelling & Software, 60, 167-179.
Citation: https://doi.org/10.5194/gmd-2021-401-RC2

Status: closed

CEC1:
'Comment on gmd-2021-401', Juan Antonio Añel, 03 Jan 2022

Dear authors,
After checking your manuscript, it has come to our attention that it does not comply with our Code and Data Policy.

https://www.geoscientific-model-development.net/policies/code_and_data_policy.html

You must address several issues:

First, you have archived your code in GitHub. However, GitHub is not a suitable repository. GitHub itself instructs authors to use other alternatives for long-term archival and publishing, such as Zenodo. Therefore, please, publish your code in one of the appropriate repositories.

Secondly, you state that the code is 'Matlab code'. This is a wrong statement. The code is 'M' code (the language). Matlab could be the interpreter you have used to run the code, but you should be aware that it could be run with other interpreters, such as the FLOSS alternative GNU Octave. Please, clarify it. If you have used Matlab, make explicit the Matlab version you have used in your work.

Also, we can not accept the ANL systems as a suitable repository to store the ELM input data. You must use one of the repositories mentioned in our policy. For example, FigShare, which you already use for other data, is suitable. The same happens with the ILAMB benchmarks.

Please, reply as soon as possible to this comment with the necessary links and DOIs for the code and data so that they are available for the peer-review process, as they should be. Also, you must include in a potential reviewed version of your manuscript the modified 'Code and Data Availability' section containing links and the DOIs.
Juan A. Añel

Geosc. Mod. Dev. Executive Editor

Citation: https://doi.org/10.5194/gmd-2021-401-CEC1
- AC1: 'Reply on CEC1', Donghui Xu, 03 Jan 2022
  
  Dear Editor,
  Thanks for reminding us regarding the Code and Data Policy, which we overlooked in our initial submission. We uploaded our ELM code, UQTk code, and processing and plotting script on Zenodo. We also clarified the version of Matlab we used to run the processing and plotting scripts. The domain data and surface data that needed for running ELM-v1, and the processed ISIMIP2a runoff benchmark are archived on Zenodo as well. We updated the Code and Data Availability section in the attached manuscript. You can also find the modified section in the following:
  Code and Data Availability
  The current version of ELM is available from E3SM project (https://github.com/E3SM-Project/E3SM/releases/tag/v1.1.0). The UQTk code and documentation are available from https://www.sandia.gov/uqtoolkit/. The exact version of ELM, exact version of UQTk source code, and scripts to produce the plots in this study is archived on Zenodo (https://doi.org/10.5281/zenodo.5815500). Matlab version R2019b Update 4 was used to run the processing and plotting scripts. ILAMB version 2 was used in this study, and the package can be accessed at 10.18139/ILAMB.v002.00/1251621. The domain file and surface data file that used to run ELMv1, and processed ISIMP2a runoff data are archived on Zenodo (https://doi.org/10.5281/zenodo.5815730). The GRUN runoff dataset was downloaded from https://doi.org/10.6084/m9.figshare.9228176.
  Thank you,
  Donghui and coauthors
  
  Citation: https://doi.org/10.5194/gmd-2021-401-AC1
RC1: 'Comment on gmd-2021-401', Anonymous Referee #1, 02 Mar 2022

This work presents a surrogate-assisted framework for calibrating runoff relevant parameters in global-scale Earth System Models (ESMs). The large computation burden arisen from repeated simulations in calibration is alleviated by building fast-to-run PCE-based surrogate models of ESMs. It is concluded that the calibrated model obtains an improved performance compared to the one with default parameter values. In summary, the manuscript is generally well-written and may be eventually accepted after addressing the following comments:

-The title should be revised. In my opinion, uncertain quantification is different from calibration. How can one use a UQ framework to calibrate models? How about “Using a surrogate-assisted Bayesian framework to ...”?

-In table 1, the prior for q_{drai,max} is U(1e-6,1e-1). Why not use a logarithmic transformation for it? Otherwise, much more prior samples will be drawn from, e.g., (1e-2, 1e-1).

-Line 193 and Eq. (17), the authors should clearly present how they determine the values of sigma.

-In section 3.3, which criterion (e.g., the Gelman-Rubin R statistic [Gelman et al., 1995]) is used here to check the convergence of MCMC sampling? From Fig. 6 it can be seen that the posterior ranges are still relatively large, which gives the feeling that the MCMC chain has not totally converged.

-In section 5.2, the poor performance of PCE surrogate models in arid regions probably because of PCE’s inability to approximate highly nonlinear functions (a well-known limitation of PCE) or/and the low signal-to-noise ratio in these regions. The authors should elaborate these to provide more informative results to readers. An alternative surrogate method for approximating highly nonlinear function is the deep neural networks.

-In Figs. 10a and 11, the simulated runoff time series with default parameter values could be even closer to the reference GRUN time series than the calibrated ones, giving the feeling that the calibration is not that satisfactory. Please further explain.

-Line 533, ‘are estimated’, Line 534, ‘are run’?

Citation: https://doi.org/10.5194/gmd-2021-401-RC1
RC2:
'Comment on gmd-2021-401', Anonymous Referee #2, 21 Mar 2022
The manuscript performed a per-grid calibration of the E3SM ELM model against a global monthly runoff dataset. The calibration was enabled by developing surrogate models for each grid of the ELM using Polynomial Chaos Expansion to mimic the response surface, which was chosen to be the root mean square error of monthly runoff for each grid. Subsequent analyses examined the spatial distribution of calibrated parameters with higher sensitivity and parametric uncertainty effects on simulated runoff. The paper is well organized, clearly written, and deals with an important topic of calibrating ELM and similar models. However, I have some concerns regarding the accuracy of surrogate and its effect on calibration, as detailed below.

Line 15: “The main methodological advance in this work is the construction of surrogates for the error metric between the ELM and the benchmark data”. But this is not entirely new as using surrogate in this manner has been done previously, e.g. Wang et al. (2014); Razavi et al. (2012) and references therein.

Line 111: what’s the difference between surface runoff and surface water runoff?

Line 195 - to reduce the log likelihood to least-squares regression, further assumption is needed, which might include constant and known sigma. Please verify.

Line 197 - I am not sure whether 1,000 samples are sufficient for burn-in, since MCMC often requires a large number (e.g., tens of thousands) of samples to converge. Including some convergence check statistics or plots in supplementary material would be helpful. Also, what is the MCMC algorithm being used here? Please include a reference for reproducibility.

Fig. 1& 2 - there’s some discrepancy between RMSE given by the surrogate and by ELM. Studies have shown that even small surrogate error can lead to large deviation of the inferred parameter posterior from the “true” posterior (Laloy and Jacques, 2019). I realize that it is not possible to calibrate ELM at global scale, but it seems possible to perform some quick test to validate the surrogate modeling approach. For example: for a few grids compare the posterior obtained using PCE and using ELM; In Section 3.5, step #4, compare the RMSE of ELM simulation with that of PCE.

Line 355: If I understand correctly, 10,000 is the number of runs of the surrogate. It is not necessarily the case if ELM is run, because the convergence rate may be different given the surrogate error (Razavi et al., 2012).

Fig. 11 - it seems that the same period of 1997-2010 is used to calibrate the model and validate the optimal parameters. Is data available after 2010 for validation, so that validation data is independent from calibration data?

Some paragraphs are indented, some are not.

References

Laloy, E., & Jacques, D. (2019). Emulation of CPU-demanding reactive transport models: a comparison of Gaussian processes, polynomial chaos expansion, and deep neural networks. Computational Geosciences, 23(5), 1193-1215.

Wang, C., Duan, Q., Gong, W., Ye, A., Di, Z., & Miao, C. (2014). An evaluation of adaptive surrogate modeling based optimization with two benchmark problems. Environmental Modelling & Software, 60, 167-179.
Citation: https://doi.org/10.5194/gmd-2021-401-RC2

Donghui Xu et al.

Supplement

https://doi.org/10.5194/gmd-2021-401-supplement

Donghui Xu et al.

Viewed

Total article views: 967 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	Supplement	BibTeX	EndNote
818	132	17	967	32	6	8

HTML: 818
PDF: 132
XML: 17
Total: 967
Supplement: 32
BibTeX: 6
EndNote: 8

Views and downloads (calculated since 16 Dec 2021)

Month	HTML	PDF	XML	Total
Dec 2021	247	37	2	286
Jan 2022	171	34	4	209
Feb 2022	78	16	2	96
Mar 2022	133	21	6	160
Apr 2022	62	12	0	74
May 2022	66	8	2	76
Jun 2022	61	4	1	66

Cumulative views and downloads (calculated since 16 Dec 2021)

Month	HTML	PDF	XML	Total
Dec 2021	247	37	2	286
Jan 2022	171	34	4	209
Feb 2022	78	16	2	96
Mar 2022	133	21	6	160
Apr 2022	62	12	0	74
May 2022	66	8	2	76
Jun 2022	61	4	1	66

Viewed (geographical distribution)

Total article views: 896 (including HTML, PDF, and XML) Thereof 896 with geography defined and 0 with unknown origin.

Country	#	Views	%

Latest update: 27 Jun 2022

Short summary

The runoff outputs in Earth System Model simulations are involved with high uncertainty, which needs to be constrained by parameter calibration. In this work, we used an Uncertainty Quantification framework to efficiently calibrate the runoff generation processes in the Energy Exascale Earth System Model V1 at global scale. The model performance was improved compared to the default parameter after calibration, and the associated parametric uncertainty was significantly constrained.


Total:	0
HTML:	0
PDF:	0
XML:	0