|Overall evaluation: The submitted manuscript entitled "How to perform global sensitivity analysis of a catchment-scale, distributed pesticide transfer model? Application to the PESHMELBA model." by Rouzies et al. applies three GSA methods to evaluate the sensitivity of the distributed process-based model to its parameters. The writing is clear and precise, and all sections are understandable. Considering the importance of such analysis for complex hydrologic models, I think the motivation and benefits of this study will be of interest to Geoscientific Model Development readers. Particularly, I like the fact that various GSA methods have been compared in this paper. That being said, the manuscript suffers from some major shortcomings with respect to its novelty and rigor. Here, I outline my comments and suggestions that should allow authors to improve their paper:|
Comment 1. The major shortcoming of the paper is that the overall value of this contribution to the hydrologic modelling community is not adequately discussed. The main contribution of this study is applying three GSA techniques to investigate the role of various parameters in pesticide transfer model. However, its merit over previous attempts is still somehow limited/not well presented. As mentioned by the authors, there are several studies where GSA approach has been applied to explore the factor importance in this context. I am not sure if this and similar studies would add much useful information to the existing body of knowledge on uncertainty analysis, parameter estimation, identifiability analysis, etc. I strongly suggest authors to clearly explain the extend to which this study is adding to the previously presented knowledge in the field (e.g., through new approaches to solving existing problems? etc.).
Having discussed the issue from that point of view, I would rather look at it from another perspective as well. Based on the reported results (Figure 7), overall, the estimated sensitivity indices by RF, HSIC, and Sobol methods are quite different. But, it is not convincing from the paper why one should use HSIC instead of Sobol or RF method. The manuscript correctly mentions the conceptual differences between three methods. For example, HSIC assesses the strength of dependencies between inputs and the output, while Sobol method attributes the variance of the output to variations in inputs or sets of inputs. However, it has not been discussed how this can help modelers/hydrologists with respect to hydrological processes’ understanding or model building. To address this issue, I strongly suggest authors provide their “objectives” and “research questions” in the introduction section by bullet points. This can properly highlight the novelty and significance of the study. Furthermore, considering the numerical results, authors should explicitly explain why and how each GSA method might be useful in the context of spatialized pesticide transfer modelling.
Comment 2. In my opinion, another major shortcoming of this paper is that there is no information about the convergence behavior of the GSA algorithms. As authors know, robust sensitivity analysis of the models typically requires many model runs, and hence considerable computational resources. So, due to the high number of model evaluations required by existing sensitivity analysis techniques and the computationally expensive nature of the models, analysts usually tend to conduct sensitivity analysis without evaluating its stability and convergence (for more discussion see, e.g., Sarrazin et al., 2016; Sheikholeslami et al., 2020). It is therefore common to choose the sample size only based on the available computational budget, which in turn can result in lack of robustness, and consequently contaminate the assessment of the sensitivities. In fact, since 5~10 years ago a surge of papers flooded the environmental modelling journals introducing/applying a sensitivity analysis technique to a model without analyzing the robustness and converges of the results. Authors should properly monitor/analyze the convergence properties of the utilized GSA techniques in identifying influential factors, for example by progressively increasing the sample size.
Comment 3. There is another important cost-effective strategy in the literature to accelerate GSA of the computationally expensive models, namely given-data approach to GSA (otherwise known as data-driven methods). To improve the literature review and strengthen the discussion part, authors can mention given-data approach in the revised manuscript. For a general review and discussion on these techniques see Sheikholeslami et al. (2021).
Comment 4. Going back to comment 3, I think an insufficient state-of-the-art has been performed in this study. There are many studies that have been previously undertaken to develop efficient screening techniques. Authors should consider existing literature in this context and perform a critical review. One notable example is the grouping approach introduced by Sheikholeslami et al. (2019). This approach uses agglomerative hierarchical clustering to categorize the parameters into distinct groups based on similarities between their sensitivity indices, and then ranks parameters according to importance group e.g., these could be labeled as “strongly influential”, “influential”, “moderately influential”, “weakly influential”, and “non-influential”) rather than individually (see Huo et al., 2019; Sheikholeslami et al., 2021 for further application of the grouping-based importance ranking approach). Other studies include Tang et al., 2007; Nossent et al., 2011; Touzani and Busby, 2014; Becker et al., 2018; etc.
Comment 5. While “parameter uncertainty” has been thoroughly analyzed in the paper, I could not find proper description about other important sources of uncertainty, particularly input data uncertainties, e.g., soil type, rain and PET forcing data. I recommend authors to add discussion regarding this important source of uncertainty which can significantly affect the model output variability. In fact, these forcings are typically the outputs of a long and complex modelling chains. Thus, PESHMELBA may simultaneously suffer from model parameter, model structure, and input uncertainties or other systematic uncertainties in precipitation bias correction, the estimation of potential evapotranspiration, or the uncertainty of deriving spatial basin scale meteorological input data.
Comment 6. Most of the existing literature on sensitivity analysis has typically been under the assumption that the controlling factors such as model parameters (processes) are independent, whereas, in many cases, they are correlated, and their joint distribution follows a variety of forms. However, very few studies in the field of water and environmental modeling address this issue. By way of example, Strobl et al. (2007) reported that when using permutation-based mean decrease in prediction accuracy as an importance measure, there might be bias in estimating importance of the correlated variables. Authors should highlight this in the revised manuscript by adding discussion on the significance of correlation effects in the utilized methods and then perhaps propose strategies (in future studies) for properly accounting for correlations in parameter (process) space.
Comment 7. I could not find any information on training and tuning of RF model. The possible inconsistency in SA results might be due to the issues in fitting RF to the input-output data. I strongly suggest authors provide details of building RF model. Furthermore, it’s not clear if RF was fitted on scalar quantities or temporal series. Without this information, results are not reliable and cannot be validated.
Comment 8. It would be interesting to see results of parameter ranking as well. Although these methods estimate different values for sensitivity indices in some cases, the ranking provided by these methods may be much more similar. Note that, in complex models, when the number of parameters is very large, we are typically not interested in an exact values of sensitivity indices. Instead, it may be more profitable to use the available computational budget to rank parameters in order of importance, e.g., “strongly influential”, “moderately influential”, and “non-influential”.
Comment 9. A possible direction for future research is to evaluate how sensitivity analysis results change by changing the selected parameter distributions (normal, log-normal, uniform,…) since there is an unavoidable uncertainty associated with defining feasible ranges of parameters.
Becker, W. E., Tarantola, S., & Deman, G. (2018). Sensitivity analysis approaches to high-dimensional screening problems at low sample size. Journal of Statistical Computation and Simulation, 88(11), 2089-2110.
Huo, X., Gupta, H., Niu, G.Y., Gong, W. and Duan, Q., 2019. Parameter sensitivity analysis for computationally intensive spatially distributed dynamical environmental systems models. Journal of Advances in Modeling Earth Systems, 11(9), pp.2896-2909.
Nossent, J., Elsen, P., & Bauwens, W. (2011). Sobol' sensitivity analysis of a complex environmental model. Environmental Modelling & Software, 26(12), 1515-1525.
Sarrazin, F., Pianosi, F., & Wagener, T. (2016). Global Sensitivity Analysis of environmental models: Convergence and validation. Environmental Modelling & Software, 79, 135-152.
Sheikholeslami, R., & Razavi, S. (2020). A fresh look at variography: measuring dependence and possible sensitivities across geophysical systems from any given data. Geophysical Research Letters, 47(20), e2020GL089829.
Sheikholeslami, R., Gharari, S., Papalexiou, S. M., & Clark, M. P. (2021). VISCOUS: A Variance-Based Sensitivity Analysis Using Copulas for Efficient Identification of Dominant Hydrological Processes. Water Resources Research, 57, e2020WR028435.
Sheikholeslami, R., Razavi, S., Gupta, H. V., Becker, W., & Haghnegahdar, A. (2019). Global sensitivity analysis for high-dimensional problems: How to objectively group factors and measure robustness and convergence while reducing computational cost. Environmental modelling & software, 111, 282-299.
Tang, Y., Reed, P., Wagener, T., & Van Werkhoven, K. (2007). Comparing sensitivity analysis methods to advance lumped watershed model identification and evaluation. Hydrol. Earth Syst. Sci., 11, 793-817.
Touzani, S., & Busby, D. (2014). Screening method using the derivative-based global sensitivity indices with application to reservoir simulator. Oil & Gas Science and Technology–Revue d’IFP Energies nouvelles, 69(4), 619-632.
Strobl, C., Boulesteix, A. L., Zeileis, A., & Hothorn, T. (2007). Bias in random forest variable importance measures: Illustrations, sources and a solution. BMC bioinformatics, 8(1), 1-21.