the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Submitted as: development and technical paper 01 Apr 2021
Submitted as: development and technical paper | 01 Apr 2021
A Model-Independent Data Assimilation (MIDA) module and its applications in ecology
- 1Center for Ecosystem Science and Society, Northern Arizona University, Flagstaff, AZ, USA
- 2School of informatics, Computing, and Cyber Systems, Northern Arizona University, Flagstaff, AZ, USA
- 3Computational Sciences and Engineering Division, Climate Change Science Institute, Oak Ridge National Laboratory, Oak Ridge, TN, USA
- 4Environmental Sciences Division, Climate Change Science Institute, Oak Ridge National Laboratory, Oak Ridge, TN, USA
- 5International Institute for Earth System Science, Nanjing University, Nanjing, China
- 6Center for Climate Systems Research, Columbia University, New York, USA
- 7NASA Goddard Institute for Space Studies, New York, USA
- 8Key Laboratory of Digital Earth Science, Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing, China
- 9Department of Biological Sciences, Northern Arizona University, Flagstaff, AZ, USA
- 1Center for Ecosystem Science and Society, Northern Arizona University, Flagstaff, AZ, USA
- 2School of informatics, Computing, and Cyber Systems, Northern Arizona University, Flagstaff, AZ, USA
- 3Computational Sciences and Engineering Division, Climate Change Science Institute, Oak Ridge National Laboratory, Oak Ridge, TN, USA
- 4Environmental Sciences Division, Climate Change Science Institute, Oak Ridge National Laboratory, Oak Ridge, TN, USA
- 5International Institute for Earth System Science, Nanjing University, Nanjing, China
- 6Center for Climate Systems Research, Columbia University, New York, USA
- 7NASA Goddard Institute for Space Studies, New York, USA
- 8Key Laboratory of Digital Earth Science, Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing, China
- 9Department of Biological Sciences, Northern Arizona University, Flagstaff, AZ, USA
Abstract. Models are an important tool to predict Earth system dynamics. An accurate prediction of future states depends on not only model structures but also parameterizations. Model parameters can be constrained by data assimilation. However, applications of data assimilation to ecology are restricted by highly technical requirements such as model-dependent coding. To alleviate this technical burden, we developed a model-independent data assimilation (MIDA) module. MIDA works in three steps including data preparation, execution of data assimilation, and visualization. The first step prepares prior ranges of parameter values, a defined number of iterations, and directory paths to access files of observations and models. The execution step calibrates parameter values to best fit the observations and estimates the parameter posterior distributions. The final step automatically visualizes the calibration performance and posterior distributions. MIDA is model independent and modelers can use MIDA for an accurate and efficient data assimilation in a simple and interactive way without modification of their original models. We applied MIDA to four types of ecological models: the data assimilation linked ecosystem carbon (DALEC) model, a surrogate-based energy exascale earth system model the land component (ELM), nine phenological models and a stand-alone biome ecological strategy simulator (BiomeE). The applications indicate that MIDA can effectively solve data assimilation problems for different ecological models. Additionally, the easy implementation and model-independent feature of MIDA breaks the technical barrier of black-box applications of data-model fusion in ecology. MIDA facilitates the assimilation of various observations into models for uncertainty reduction in ecological modeling and forecasting.
Xin Huang et al.
Status: closed
-
RC1: 'Comment on gmd-2021-33', Anonymous Referee #1, 27 Apr 2021
Review of “A model independent data assimilation (MIDA) module and its applications in ecology”
Authors: Huang et al.,
Summary
The authors identify the growing role data assimilation is playing in reducing parametrisation errors / uncertainty thus supporting data and/or model structural selection for terrestrial ecosystem models to improve predictive skill. The authors also identify an important obstacle to wider uptake is data assimilation frameworks being built around the requirements of a specific model, i.e. they are not easily applicable to alternate models. In response the authors have created a data assimilation framework which aims to be model-independent reducing the coding barrier.
Overall I don’t have any objections to this paper being published more or less as is. The Github repository appears well organised with code that is commented to help people learn the framework. I would like a little more detail of is contained within the “black box” placed in the SI to give those who are interested this information.
Introduction
The introduction make a clear description of the state of data assimilation applications in ecology and identifies a key obstacle (i.e. substantial programming skill / investment in time) to wider engagement within the ecological community.
Methods
A clear description of the process MIDA is given. I wonder about the trade-offs between the usually very fast exchange of information achieved when writing an interface vs the more user friendly approach described here? Not an objection to your approach but genuinely curious.
What isn’t quite so clear is the details of how MIDA knows which information in the existing model output files corresponds to its observations. For example, the namelist.txt must contain information on the variable names used to describe the observations and their corresponding output variable generated by the model? These must still vary depending on the model being used? A screen shot showing the interface which is populated with an example would make this really clear.
On a similar point. The models need to be able to read the parameters from a file. The MIDA framework must then be able to write out the proposed parameters in a unique format for each model, is that correct?
L322-330: Could you add a link to further details in SI for this section? The reason I ask is that Haario et al., (2001) steps based on the weighted (e.g. beta) combination of the multivariate Gaussian and a minimum step size scaled by a value drawn from a Gaussian distribution of mean = 0, sd = 1. The multivariate Gaussian being derived from the covariance matrix for the parameters adjusted by an optimal scaling parameter (e.g. 2.38 / npars^0.5). The weighting between the two steps (beta ~0.05) and the minimum step size. So which of these variables (or something else entirely) for example is you “jump scaling”?
I like the inclusion of a screenshot of the software but I think it would be useful to have an example which has been filled in to help guide the potential user. Alternatively showing an example of the namelist.txt might be informative.
Applications with MIDA
Case 1: This doesn’t really impact the validity of the paper but just something I noticed and wanted to raise as it should really be clarified. The DALEC model is stated as having 5 C pools but also to having a Growing Degree Days phenology model. However, the Williams et al., (2005) model doesn’t have phenology model (i.e. continuous allocation / evergreen). DALEC was split into deciduous and evergreen versions in Fox et al., (2009) as part of the reflex project adding a 6th pool and the GDD model. The example DALEC code provided on the MIDA Github shows a alternate version of the model where leaf C is not dependent on GPP (and thus the system is not mass balanced). This is a distraction from the main point of demonstrating your DA system. Please make the origin of the code clear as it doesn’t match that found in the citations given. For example, the paper cited with the 5 pools Ricciuto et al., (2011) describes a model called LoTEC. Is this a DALEC derivative?
Case 3: It is really nice to have the comparison between multiple hypothesised model structures tested here. I think this is a really nice demonstration of the sort of simple experiments that can be achieved but are really useful.
Discussions
No comment, really happy.
Technical comments
L140: “DA is a statistical approach...” – there are many different algorithms for DA whether for state update or parameter estimation (in this case). I think it would be clearer refer to it as an “approach”. I can see that you are trying to talk about your specific approach so maybe "The DA approach embeded within MIDA..."
L176: “hinders” or “hides”?
L454: “This model simulates..."
-
AC2: 'Reply on RC1', Xin Huang, 04 Jun 2021
Dear reviewer,
Thanks very much for taking your time to review this manuscript. We really appreciate all your comments and suggestions, which are very helpful for us to improve the manuscript. We have made a thorough revision to address the comments and questions. Please find our point-to-point responses in the attachment.
-
AC2: 'Reply on RC1', Xin Huang, 04 Jun 2021
-
RC2: 'Comment on gmd-2021-33', Anonymous Referee #2, 11 May 2021
Summary
Huang et al created a model-independent data assimilation module and applied their module to different types of applications in ecology. This article is timely as data assimilation has become an urgent need but remains difficult for ecologists to implement. The details of implementation are extensive, but the benefits of the approach are somewhat repetitive throughout the manuscript. This type of tool is of great use to many types of modelers in ecology who don’t want to invest the time to add their model to a system. The details and video are very helpful for non-specialists in model calibration. I think there should be more papers like this so there can be more research using data assimilation techniques. However, I do have some concerns about MIDA which are described below. I think this paper should be published with revisions addressing Section 2.4.
General Comments
I have made several specific comments that are listed below. My three major comments are 1) that the statistical model used for model calibration in Section 2.4 “Step: Execution of data assimilation” is not well defined. It’s quite unclear to me what the authors are using for model calibration therefore it is difficult to evaluate the calibration exercises themselves. I’m not convinced that you are really showing posterior distributions in the figures because of the description of the algorithm in the methods. What is the actual statistical model used for model calibration? More details on my questions about this section are in my specific comments section 2) My second major comment is that Fer 2018 and PEcAn were entirely left out of this manuscript but should certainly be included in several places.
Specific Comments
Line 26: what kinds of states?
Line 38: I think there’s a word missing “model … the land component”
Line 42: Doesn’t the easy implementation potentially make this more ‘black box’?
Line 58: Citation for invasive coding?
Line 75: Missing link sentence between “, 2009). … DA was”
Line 78: Not sure about “data-worth”
Line 100: Include Predictive Ecosystem Analyzer (PEcAn)
Line 120-121: Not sure what is meant by this sentence
Line 131: Nice!
Line 176: Hinders?
Line 178: How does MIDA know how to write out model specific configurations?
Line 183 transfers -> transfer
Line 184: organize -> organizes
Line 184: So the “different files” are like lookup tables?
Line 194: Would be cool to have an illustration of dynamic initialization *
Line 209: In think you mean “inference”
Line 210: Before talking about the sampling algorithm, it would be good for the reader to know about the model formulation like the likelihood, prior, etc.
Line 245: What kind of MLE? What’s the model formulation? What are you maximizing over here? Why do you get maximum likelihoods and posteriors?
Line 250: At this point, I’m still confused how you define your drivers and model settings in general? For a particular model you usually have a parameter file that gets read by the executable. So are you rewriting those parameter files in step 2?
Line 263: Make sure to cite all software packages.
Line 314: Helpful examples. Really curious where 302 years of leaf area data come from!
Line 343: “complex reasons” is somewhat vague
Line 499: Citation?
Line 504: Not sure about “first” unless you make your definition of model agnostic more specific
Figure 1: Very nice!
Figure 6: the Cis appear homogenous. Can MIRA deal with heteroskedasticity?
Figure 7: the colors in the legend have an extra character that should be removed
Figure 4, 5, and 8: these posterior distributions do not look converged to me. Could you also include the mcmc chains as well here or in the supplements?
-
AC3: 'Reply on RC2', Xin Huang, 04 Jun 2021
Dear reviewer,
Thanks very much for taking your time to review this manuscript. We really appreciate all your comments and suggestions, which are very helpful for us to improve the manuscript. We have made a thorough revision to address the comments and questions. Please find our point-to-point responses attached.
-
AC3: 'Reply on RC2', Xin Huang, 04 Jun 2021
-
CEC1: 'Comment on gmd-2021-33', Juan Antonio Añel, 14 May 2021
Dear authors,
Unfortunately, as our journal's Code and Data Policy establishes, we can not accept Github as a trustful repository for the code and data you use in your manuscript. GitHub itself says it and recommends Zenodo for long-term code and data archival.
Therefore, please, check our policy and upload your materials to a suitable repository. And be aware that we can not publish your manuscript or move to the next steps in the review proccess until you fix it.https://www.geoscientific-model-development.net/policies/code_and_data_policy.html#item3
Regards,
Dr. juan A. Añel
Geosc. Mod. Dev. Executive Editor
-
AC1: 'Reply on CEC1', Xin Huang, 14 May 2021
Thanks for the notification. I uploaded code and data to Zenodo instead of GitHub. Please find the DOIs below.
Code DOI: 10.5281/zenodo.4762725 Target URL: https://doi.org/10.5281/zenodo.4762725
Video DOI: 10.5281/zenodo.4762777 Target URL: https://doi.org/10.5281/zenodo.4762777
Data DOI: 10.5281/zenodo.4762779 Target URL: https://doi.org/10.5281/zenodo.4762779
If there is anything wrong, please let me know. Thanks again.
-
AC1: 'Reply on CEC1', Xin Huang, 14 May 2021
Status: closed
-
RC1: 'Comment on gmd-2021-33', Anonymous Referee #1, 27 Apr 2021
Review of “A model independent data assimilation (MIDA) module and its applications in ecology”
Authors: Huang et al.,
Summary
The authors identify the growing role data assimilation is playing in reducing parametrisation errors / uncertainty thus supporting data and/or model structural selection for terrestrial ecosystem models to improve predictive skill. The authors also identify an important obstacle to wider uptake is data assimilation frameworks being built around the requirements of a specific model, i.e. they are not easily applicable to alternate models. In response the authors have created a data assimilation framework which aims to be model-independent reducing the coding barrier.
Overall I don’t have any objections to this paper being published more or less as is. The Github repository appears well organised with code that is commented to help people learn the framework. I would like a little more detail of is contained within the “black box” placed in the SI to give those who are interested this information.
Introduction
The introduction make a clear description of the state of data assimilation applications in ecology and identifies a key obstacle (i.e. substantial programming skill / investment in time) to wider engagement within the ecological community.
Methods
A clear description of the process MIDA is given. I wonder about the trade-offs between the usually very fast exchange of information achieved when writing an interface vs the more user friendly approach described here? Not an objection to your approach but genuinely curious.
What isn’t quite so clear is the details of how MIDA knows which information in the existing model output files corresponds to its observations. For example, the namelist.txt must contain information on the variable names used to describe the observations and their corresponding output variable generated by the model? These must still vary depending on the model being used? A screen shot showing the interface which is populated with an example would make this really clear.
On a similar point. The models need to be able to read the parameters from a file. The MIDA framework must then be able to write out the proposed parameters in a unique format for each model, is that correct?
L322-330: Could you add a link to further details in SI for this section? The reason I ask is that Haario et al., (2001) steps based on the weighted (e.g. beta) combination of the multivariate Gaussian and a minimum step size scaled by a value drawn from a Gaussian distribution of mean = 0, sd = 1. The multivariate Gaussian being derived from the covariance matrix for the parameters adjusted by an optimal scaling parameter (e.g. 2.38 / npars^0.5). The weighting between the two steps (beta ~0.05) and the minimum step size. So which of these variables (or something else entirely) for example is you “jump scaling”?
I like the inclusion of a screenshot of the software but I think it would be useful to have an example which has been filled in to help guide the potential user. Alternatively showing an example of the namelist.txt might be informative.
Applications with MIDA
Case 1: This doesn’t really impact the validity of the paper but just something I noticed and wanted to raise as it should really be clarified. The DALEC model is stated as having 5 C pools but also to having a Growing Degree Days phenology model. However, the Williams et al., (2005) model doesn’t have phenology model (i.e. continuous allocation / evergreen). DALEC was split into deciduous and evergreen versions in Fox et al., (2009) as part of the reflex project adding a 6th pool and the GDD model. The example DALEC code provided on the MIDA Github shows a alternate version of the model where leaf C is not dependent on GPP (and thus the system is not mass balanced). This is a distraction from the main point of demonstrating your DA system. Please make the origin of the code clear as it doesn’t match that found in the citations given. For example, the paper cited with the 5 pools Ricciuto et al., (2011) describes a model called LoTEC. Is this a DALEC derivative?
Case 3: It is really nice to have the comparison between multiple hypothesised model structures tested here. I think this is a really nice demonstration of the sort of simple experiments that can be achieved but are really useful.
Discussions
No comment, really happy.
Technical comments
L140: “DA is a statistical approach...” – there are many different algorithms for DA whether for state update or parameter estimation (in this case). I think it would be clearer refer to it as an “approach”. I can see that you are trying to talk about your specific approach so maybe "The DA approach embeded within MIDA..."
L176: “hinders” or “hides”?
L454: “This model simulates..."
-
AC2: 'Reply on RC1', Xin Huang, 04 Jun 2021
Dear reviewer,
Thanks very much for taking your time to review this manuscript. We really appreciate all your comments and suggestions, which are very helpful for us to improve the manuscript. We have made a thorough revision to address the comments and questions. Please find our point-to-point responses in the attachment.
-
AC2: 'Reply on RC1', Xin Huang, 04 Jun 2021
-
RC2: 'Comment on gmd-2021-33', Anonymous Referee #2, 11 May 2021
Summary
Huang et al created a model-independent data assimilation module and applied their module to different types of applications in ecology. This article is timely as data assimilation has become an urgent need but remains difficult for ecologists to implement. The details of implementation are extensive, but the benefits of the approach are somewhat repetitive throughout the manuscript. This type of tool is of great use to many types of modelers in ecology who don’t want to invest the time to add their model to a system. The details and video are very helpful for non-specialists in model calibration. I think there should be more papers like this so there can be more research using data assimilation techniques. However, I do have some concerns about MIDA which are described below. I think this paper should be published with revisions addressing Section 2.4.
General Comments
I have made several specific comments that are listed below. My three major comments are 1) that the statistical model used for model calibration in Section 2.4 “Step: Execution of data assimilation” is not well defined. It’s quite unclear to me what the authors are using for model calibration therefore it is difficult to evaluate the calibration exercises themselves. I’m not convinced that you are really showing posterior distributions in the figures because of the description of the algorithm in the methods. What is the actual statistical model used for model calibration? More details on my questions about this section are in my specific comments section 2) My second major comment is that Fer 2018 and PEcAn were entirely left out of this manuscript but should certainly be included in several places.
Specific Comments
Line 26: what kinds of states?
Line 38: I think there’s a word missing “model … the land component”
Line 42: Doesn’t the easy implementation potentially make this more ‘black box’?
Line 58: Citation for invasive coding?
Line 75: Missing link sentence between “, 2009). … DA was”
Line 78: Not sure about “data-worth”
Line 100: Include Predictive Ecosystem Analyzer (PEcAn)
Line 120-121: Not sure what is meant by this sentence
Line 131: Nice!
Line 176: Hinders?
Line 178: How does MIDA know how to write out model specific configurations?
Line 183 transfers -> transfer
Line 184: organize -> organizes
Line 184: So the “different files” are like lookup tables?
Line 194: Would be cool to have an illustration of dynamic initialization *
Line 209: In think you mean “inference”
Line 210: Before talking about the sampling algorithm, it would be good for the reader to know about the model formulation like the likelihood, prior, etc.
Line 245: What kind of MLE? What’s the model formulation? What are you maximizing over here? Why do you get maximum likelihoods and posteriors?
Line 250: At this point, I’m still confused how you define your drivers and model settings in general? For a particular model you usually have a parameter file that gets read by the executable. So are you rewriting those parameter files in step 2?
Line 263: Make sure to cite all software packages.
Line 314: Helpful examples. Really curious where 302 years of leaf area data come from!
Line 343: “complex reasons” is somewhat vague
Line 499: Citation?
Line 504: Not sure about “first” unless you make your definition of model agnostic more specific
Figure 1: Very nice!
Figure 6: the Cis appear homogenous. Can MIRA deal with heteroskedasticity?
Figure 7: the colors in the legend have an extra character that should be removed
Figure 4, 5, and 8: these posterior distributions do not look converged to me. Could you also include the mcmc chains as well here or in the supplements?
-
AC3: 'Reply on RC2', Xin Huang, 04 Jun 2021
Dear reviewer,
Thanks very much for taking your time to review this manuscript. We really appreciate all your comments and suggestions, which are very helpful for us to improve the manuscript. We have made a thorough revision to address the comments and questions. Please find our point-to-point responses attached.
-
AC3: 'Reply on RC2', Xin Huang, 04 Jun 2021
-
CEC1: 'Comment on gmd-2021-33', Juan Antonio Añel, 14 May 2021
Dear authors,
Unfortunately, as our journal's Code and Data Policy establishes, we can not accept Github as a trustful repository for the code and data you use in your manuscript. GitHub itself says it and recommends Zenodo for long-term code and data archival.
Therefore, please, check our policy and upload your materials to a suitable repository. And be aware that we can not publish your manuscript or move to the next steps in the review proccess until you fix it.https://www.geoscientific-model-development.net/policies/code_and_data_policy.html#item3
Regards,
Dr. juan A. Añel
Geosc. Mod. Dev. Executive Editor
-
AC1: 'Reply on CEC1', Xin Huang, 14 May 2021
Thanks for the notification. I uploaded code and data to Zenodo instead of GitHub. Please find the DOIs below.
Code DOI: 10.5281/zenodo.4762725 Target URL: https://doi.org/10.5281/zenodo.4762725
Video DOI: 10.5281/zenodo.4762777 Target URL: https://doi.org/10.5281/zenodo.4762777
Data DOI: 10.5281/zenodo.4762779 Target URL: https://doi.org/10.5281/zenodo.4762779
If there is anything wrong, please let me know. Thanks again.
-
AC1: 'Reply on CEC1', Xin Huang, 14 May 2021
Xin Huang et al.
Xin Huang et al.
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 449 | 149 | 16 | 614 | 3 | 8 |
- HTML: 449
- PDF: 149
- XML: 16
- Total: 614
- BibTeX: 3
- EndNote: 8
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1