Review of “A model independent data assimilation (MIDA) module and its applications in ecology”

Authors: Huang et al.,

Summary

The authors identify the growing role data assimilation is playing in reducing parametrisation errors / uncertainty thus supporting data and/or model structural selection for terrestrial ecosystem models to improve predictive skill. The authors also identify an important obstacle to wider uptake is data assimilation frameworks being built around the requirements of a specific model, i.e. they are not easily applicable to alternate models. In response the authors have created a data assimilation framework which aims to be model-independent reducing the coding barrier.

Overall I don’t have any objections to this paper being published more or less as is. The Github repository appears well organised with code that is commented to help people learn the framework. I would like a little more detail of is contained within the “black box” placed in the SI to give those who are interested this information.

Introduction

The introduction make a clear description of the state of data assimilation applications in ecology and identifies a key obstacle (i.e. substantial programming skill / investment in time) to wider engagement within the ecological community.

Methods

A clear description of the process MIDA is given. I wonder about the trade-offs between the usually very fast exchange of information achieved when writing an interface vs the more user friendly approach described here? Not an objection to your approach but genuinely curious.

What isn’t quite so clear is the details of how MIDA knows which information in the existing model output files corresponds to its observations. For example, the namelist.txt must contain information on the variable names used to describe the observations and their corresponding output variable generated by the model? These must still vary depending on the model being used? A screen shot showing the interface which is populated with an example would make this really clear.

On a similar point. The models need to be able to read the parameters from a file. The MIDA framework must then be able to write out the proposed parameters in a unique format for each model, is that correct?

L322-330: Could you add a link to further details in SI for this section? The reason I ask is that Haario et al., (2001) steps based on the weighted (e.g. beta) combination of the multivariate Gaussian and a minimum step size scaled by a value drawn from a Gaussian distribution of mean = 0, sd = 1. The multivariate Gaussian being derived from the covariance matrix for the parameters adjusted by an optimal scaling parameter (e.g. 2.38 / npars^0.5). The weighting between the two steps (beta ~0.05) and the minimum step size. So which of these variables (or something else entirely) for example is you “jump scaling”?

I like the inclusion of a screenshot of the software but I think it would be useful to have an example which has been filled in to help guide the potential user. Alternatively showing an example of the namelist.txt might be informative.

Applications with MIDA

Case 1: This doesn’t really impact the validity of the paper but just something I noticed and wanted to raise as it should really be clarified. The DALEC model is stated as having 5 C pools but also to having a Growing Degree Days phenology model. However, the Williams et al., (2005) model doesn’t have phenology model (i.e. continuous allocation / evergreen). DALEC was split into deciduous and evergreen versions in Fox et al., (2009) as part of the reflex project adding a 6^th pool and the GDD model. The example DALEC code provided on the MIDA Github shows a alternate version of the model where leaf C is not dependent on GPP (and thus the system is not mass balanced). This is a distraction from the main point of demonstrating your DA system. Please make the origin of the code clear as it doesn’t match that found in the citations given. For example, the paper cited with the 5 pools Ricciuto et al., (2011) describes a model called LoTEC. Is this a DALEC derivative?

Case 3: It is really nice to have the comparison between multiple hypothesised model structures tested here. I think this is a really nice demonstration of the sort of simple experiments that can be achieved but are really useful.

Discussions

No comment, really happy.

Technical comments

L140: “DA is a statistical approach...” – there are many different algorithms for DA whether for state update or parameter estimation (in this case). I think it would be clearer refer to it as an “approach”. I can see that you are trying to talk about your specific approach so maybe "The DA approach embeded within MIDA..."

L176: “hinders” or “hides”?

L454: “This model simulates..."

Summary

Huang et al created a model-independent data assimilation module and applied their module to different types of applications in ecology. This article is timely as data assimilation has become an urgent need but remains difficult for ecologists to implement. The details of implementation are extensive, but the benefits of the approach are somewhat repetitive throughout the manuscript. This type of tool is of great use to many types of modelers in ecology who don’t want to invest the time to add their model to a system. The details and video are very helpful for non-specialists in model calibration. I think there should be more papers like this so there can be more research using data assimilation techniques. However, I do have some concerns about MIDA which are described below. I think this paper should be published with revisions addressing Section 2.4.

General Comments

I have made several specific comments that are listed below. My three major comments are 1) that the statistical model used for model calibration in Section 2.4 “Step: Execution of data assimilation” is not well defined. It’s quite unclear to me what the authors are using for model calibration therefore it is difficult to evaluate the calibration exercises themselves. I’m not convinced that you are really showing posterior distributions in the figures because of the description of the algorithm in the methods. What is the actual statistical model used for model calibration? More details on my questions about this section are in my specific comments section 2) My second major comment is that Fer 2018 and PEcAn were entirely left out of this manuscript but should certainly be included in several places.

Specific Comments

Line 26: what kinds of states?

Line 38: I think there’s a word missing “model … the land component”

Line 42: Doesn’t the easy implementation potentially make this more ‘black box’?

Line 58: Citation for invasive coding?

Line 75: Missing link sentence between “, 2009). … DA was”

Line 78: Not sure about “data-worth”

Line 100: Include Predictive Ecosystem Analyzer (PEcAn)

Line 120-121: Not sure what is meant by this sentence

Line 131: Nice!

Line 176: Hinders?

Line 178: How does MIDA know how to write out model specific configurations?

Line 183 transfers -> transfer

Line 184: organize -> organizes

Line 184: So the “different files” are like lookup tables?

Line 194: Would be cool to have an illustration of dynamic initialization *

Line 209: In think you mean “inference”

Line 210: Before talking about the sampling algorithm, it would be good for the reader to know about the model formulation like the likelihood, prior, etc.

Line 245: What kind of MLE? What’s the model formulation? What are you maximizing over here? Why do you get maximum likelihoods and posteriors?

Line 250: At this point, I’m still confused how you define your drivers and model settings in general? For a particular model you usually have a parameter file that gets read by the executable. So are you rewriting those parameter files in step 2?

Line 263: Make sure to cite all software packages.

Line 314: Helpful examples. Really curious where 302 years of leaf area data come from!

Line 343: “complex reasons” is somewhat vague

Line 499: Citation?

Line 504: Not sure about “first” unless you make your definition of model agnostic more specific

Figure 1: Very nice!

Figure 6: the Cis appear homogenous. Can MIRA deal with heteroskedasticity?

Figure 7: the colors in the legend have an extra character that should be removed

Figure 4, 5, and 8: these posterior distributions do not look converged to me. Could you also include the mcmc chains as well here or in the supplements?