the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 30 May 2024
A Unified System for Evaluating, Ranking and Clustering in Diverse Scientific Domains
Abstract. Evaluating, ranking, and clustering (ERC) stand as fundamental tasks in scientific research, each requiring a mathematical foundation. This study presents an ERC system anchored in the CCHZ-DISO (Chen, Chen, Hu, and Zhou-Distance between Indices of Simulation and Observation) system. Previous research underscores the optimality achieved by the CCHZ-DISO system (Hu et al., 2022). Since the inception of CCHZ- DISO-series research by Hu et al. (2019), DISO has found extensive applications across various domains including geography, hydrology, and economics. Analogous to the CCHZ-DISO system's construction, the ERC system employs the Euclidean distance to perform evaluating, ranking, and clustering tasks. Furthermore, illustrative examples are provided to elucidate the application of the ERC system. In fact, the ERC system unified the evaluating, ranking, and clustering tasks in one simple equation which is more flexible and simpler than the present system. It will have a more widely application than CCHZ-DISO in diverse scientific domains.
This preprint has been withdrawn.
-
Withdrawal notice
This preprint has been withdrawn.
-
Preprint
(2242 KB)
-
Supplement
(126 KB)
-
This preprint has been withdrawn.
- Preprint
(2242 KB) - Metadata XML
-
Supplement
(126 KB) - BibTeX
- EndNote
Interactive discussion
Status: closed
-
RC1: 'Comment on gmd-2024-82', Anonymous Referee #1, 21 Jun 2024
This study designed a tool for Evaluation, Ranking, and Clustering (ERC) tasks based on Euclidean distance. The main issues are as follows:
- What is the importance and necessity of using a common math mathematical framework to complete Evaluation, Ranking, and Clustering (ERC) tasks from a scientific perspective? The discussion in this research is not sufficient.
- The method established in this paper essentially uses Euclidean distance for evaluation, ranking, and clustering. If there have been any evaluation method based on Euclidean distance for the three fields mentioned above? This study lacks a comprehensive review and summary of existing methods.
- What are the advantages of the method proposed in this study compared to existing methods? What is the framework for evaluating different methods? And how can the superiority of this method be scientifically proven? There is still insufficient work in this study to address the aforementioned issues.
Citation: https://doi.org/10.5194/gmd-2024-82-RC1 -
AC1: 'Reply on RC1', Zengyun Hu, 23 Jun 2024
Comment 1: What is the importance and necessity of using a common math mathematical framework to complete Evaluation, Ranking, and Clustering (ERC) tasks from a scientific perspective? The discussion in this research is not sufficient.
Reply Thanks for your good suggestion. In general, a common mathematical model can be widely applied in numerous scientific domains due to its advantage in revealing the essential of the change law in things. Especially, the common mathematical models usually have the advantages applied in the interdisciplinary areas.
The present models/approaches about Evaluation, Ranking, and Clustering always focus on some special research areas. In essential, they completely can be unified by a common mathematical model since they have the same Euclidean distance characteristics.
Moreover, the ERC system proposed in our study includes some advantages than present Evaluation, Ranking, and Clustering models, which have been illustrated in our manuscript. Therefore, it is very necessary and urgent to unified the present complex and various models about Evaluation, Ranking, and Clustering in a common mathematical model: ERC model/ ERC system.
Comment 2: The method established in this paper essentially uses Euclidean distance for evaluation, ranking, and clustering. If there has been any evaluation method based on Euclidean distance for the three fields mentioned above? This study lacks a comprehensive review and summary of existing methods.
Reply Thanks for your good suggestion. In fact, we also want to search some methods to address the three fields of Evaluation, Ranking, and Clustering. Unfortunately, there is no unified method for the three fields. That is why we propose our ERC system to address the three fields.
Previous studies, such as Taylor diagram (Taylor 2001) Nash-Sutcliffe efficiency (NSE) (Nash and Sutcliffe, 1970) and Kling-Gupta efficiency coefficient (KGE) (Gupta et al., 2008), only employed limited statistical metrics and their applications only focus some special research areas. They are not widely applied in various departments.
References
Gupta, H., Kling, H., Yilmaz, K., and Martinez, G., Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling, Journal of Hydrology, 377, 80-91, doi:10.1016/j.jhydrol.2009.08.003, 2009.
Nash, J.E., Sutcliffe, J.V., River flow forecasting through. Part I. A conceptual models discussion of principles. Journal of Hydrology. 10, 282-290, 1970.
Taylor, K. Summarizing multiple aspects of model performance in a single diagram. Journal of Geophysical Research, 106, 7183-7192, 2001.
Comment 3: What are the advantages of the method proposed in this study compared to existing methods? What is the framework for evaluating different methods? And how can the superiority of this method be scientifically proven? There is still insufficient work in this study to address the aforementioned issues.
Reply Thanks for your good suggestions. The ERC System proposed in this manuscript is our series research of DISO (Distance between Indices of Simulation and Observation) (Hu et al., 2019, 2022; Zhou et al., 2021). ERC not only contains all the advantages of DISO, but also is extended to Ranking, and Clustering. The advantages of ERC System are provided as follows.
1 The dimension of ERC is from one to infinity, which is more flexible and simpler than present system.
2 It can include all the statistical metrics, not some special metrics in other models.
3 It successfully solves the multiple variables, multiple weights in the three fields.
The framework for evaluating different methods is Euclidean distance.
The Third-Party Evaluations of DISO can objective and impartial to show ERC’s advantage as follows.
Since its initial publication, CCHZ-DISO has garnered significant traction and witnessed widespread application, garnering over 100 citations in the Web of Science within the past three years. In this section, we showcase a selection of notable objective and positive evaluations from third-party sources, underscoring the impact and utility of the CCHZ-DISO system.
Kalmar et al. (2021) first recommended the DISO index for assessing historical regional precipitation simulations with the RegCM4.5 model. They compared DISO and Taylor diagram methods and found that “The advantage of using DISO versus the Taylor diagram is that the comprehensive performances of the different models are still not quantified by the latter”. In a sensitivity analysis of soil water and heat transfer parameters in community land surface models, Deng et al. (2021) introduced DISO, as discussed in section 2.3 of their paper published in the “Journal of Advances in Modeling Earth Systems”. Moreover, they declared that “In this paper, the most important and best advantage of DISO is that after normalizing the observed and simulated data, the value of DISO can express the performance of the same model at different sites”.
Wu et al. (2023) suggested that DISO has more advantages than Taylor diagrams, noting that “DISO overcomes some disadvantages of Taylor diagrams and provides an intuitive way to measure differences between various GCMs in the same assessment system”; additionally, the limitations of Taylor diagrams were discussed: “Taylor diagrams (Taylor 2001) are the most common way to assess the performance of climate products. However, they inevitably have inherent drawbacks…. The DISO (Hu et al. 2019) algorithm was designed to overcome the drawbacks that exist in Taylor diagrams. First, DISO has a higher dimensionality than Taylor diagrams… In addition, DISO can evaluate the performance of a model based on multiple metrics at the same time, which can reflect the performance of the model in different aspects”. The comprehensive performance of the CCHZ-DISO has been explored in many studies (Zhuang et al., 2023; Liu et al., 2022; Longo-Minnolo et al., 2022; Ma et al., 2022; Yin et al., 2022).
The third-party evaluations unequivocally indicate that, in comparison to Taylor diagrams, CCHZ-DISO exhibits superior advantages. It stands as an efficient and highly effective approach for the comprehensive quantification of performance across diverse models, providing a holistic assessment of their overall capabilities.
References:
Deng, M., Meng, X. and Lu, Y., et al., 2021, Impact and Sensitivity Analysis of Soil Water and Heat Transfer Parameterizations in Community Land Surface Model on the Tibetan Plateau, Journal of Advances in Modeling Earth Systems, 13, e2021MS002670.
Kalmar, T., Pieczka, H. and Pongracz, R., 2021, A sensitivity analysis of the different setups of the RegCM 4.5 model for the Carpathian region, International Journal of Climatology, 41, E1180-E1201,
Hu, Z., Chen, X. and Zhou, Q., et al., 2019, DISO: A rethink of Taylor diagram. International Journal of Climatology, 39(5), 2825-2832.
Liu, Z., Huang, J. and Xiao, X., et al., 2022, The capability of CMIP6 models on seasonal precipitation extremes over Central Asia, Atmospheric Research, 278, 106364.
Longo-Minnolo, G., Vanella, D. and Consoli, S., et al., 2022, Assessing the use of ERA5-Land reanalysis and spatial interpolation methods for retrieving precipitation estimates at basin scale, Atmospheric Research, 271, 106131.
Ma, R., Xiao, J. and Liang, S., 2022, Pixel-level parameter optimization of a terrestrial biosphere model for improving estimation of carbon fluxes with an efficient model-data fusion method and satellite-derived LAI and GPP data, Geoscientific Model Development, 15, 6637-6657.
Taylor, K., 2001, Summarizing multiple aspects of model performance in a single diagram. Journal of Geophysical Research, 106, 7183-7192.
Citation: https://doi.org/10.5194/gmd-2024-82-AC1
-
RC2: 'Comment on gmd-2024-82', Anonymous Referee #2, 25 Jun 2024
The manuscript proposed an ERC system anchored in the CCHZ-DISO (Chen, Chen, Hu, and Zhou-Distance between Indices of Simulation and Observation) system. Analogous to the CCHZ-DISO system's construction, the ERC system employs the Euclidean distance to perform evaluating, ranking, and clustering tasks. Furthermore, three examples are provided to elucidate the application of the ERC system. In general, the structure of the manuscript is clear, and the expression is smooth. However, the biggest issue with the article is a lack of innovation. Readers will wonder what is truly new and what is particularly special. As one of the principal criteria in peer review, the scientific significance of this manuscript is Poor. It is crucial for the authors to provide a thorough justification for the novelty of their work and to clearly differentiate it from existing research and systems.
- The manuscript claims to present a unified system for evaluating, ranking, and clustering (ERC) based on the CCHZ-DISO system. The novelty would need to be assessed in terms of whether the integration of these three tasks into one system is indeed a new approach or if similar systems have been proposed before.
- The use of Euclidean distance as the mathematical foundation for the ERC tasks is presented as a simple yet potentially powerful method. The innovation here could be in how it is applied across different scientific domains, but it needs to be clear if this application is unique.
- The paper suggests that the ERC system is versatile and can be applied to various scientific fields. The innovation might lie in its adaptability, but it is important to scrutinize whether this cross-domain applicability is truly novel or if other systems have demonstrated similar flexibility. The manuscript posits that the ERC system simplifies complex tasks. Innovation could be in the simplification process itself, but it must be determined if this simplification is indeed novel or if it is a reiteration of existing methods. In section 5, here would be best to have a table comparing existing similar systems, as well as their advantages and disadvantages.
- To assert the novelty of the ERC system, it would be necessary to compare it with existing evaluation, ranking, and clustering methods. The manuscript should clearly articulate why and how the ERC system is different and potentially superior to these methods. In applications in section 6, the evaluation results should include comparisons with other systems. Or, at least, comparisons with other similar assessment systems should be discussed.
- If the manuscript introduces new theoretical concepts or frameworks that underpin the ERC system, these should be clearly outlined and compared with existing theories to highlight their novelty. Similarly, if the innovation lies in the practical application of the system, such as improved efficiency, accuracy, or ease of use, these benefits should be clearly demonstrated and compared with the existing state of the art. The manuscript should provide a clear explanation of why the approach taken is original, including any unique methodologies, algorithms, or data uses that have not been explored in previous research.
- In regard to the applicability of DISO, this paper should especially clarify that the majority of dimensionless measurement indices targeting multiple objectives (or different physical quantities) tend to create a significant amount of ineffective search spaces or paths, thus rendering them less suitable for the field such as the bionic or meta-heuristic artificial intelligence algorithms.
- The authors mentioned the flexibility in selecting statistical metrics for the ERC system. However, it would be helpful to provide guidance on the criteria for selecting these metrics in the context of geoscientific applications, where data characteristics can be quite varied.
- The discussion on the comparison between NSE, KGE, and CCHZ-DISO is appreciated. However, it would be advantageous to include a more comprehensive comparison with other prevalent methods in geosciences, such as the Taylor diagram (and does the DISO’s advantage is simpler?), to position the ERC system within the field better.
- The paper briefly touches on the significance testing for models with small differences in ERC values. It would be valuable to see a more detailed explanation of how this testing is conducted and its implications for geoscientific studies, where subtle differences can be critical.
- The authors state that the ERC system does not consider data characteristics such as outliers. Given the common presence of outliers in geoscience data, it is important to discuss how the ERC system's results might be affected and whether any adjustments or preprocessing are recommended.
- Technical corrections, i.e.,
In Line 340, repeated “clustering”
Figure 3, the position of its reference to is inaccurate, and all abbreviations of models should be given their full names, as well as brief introductions of different models, for broader reading.
Citation: https://doi.org/10.5194/gmd-2024-82-RC2 - AC2: 'Reply on RC2', Zengyun Hu, 03 Jul 2024
-
RC3: 'Comment on gmd-2024-82', Anonymous Referee #3, 08 Jul 2024
General comments:
In the manuscript titled “A Unified System for Evaluating, Ranking and Clustering in Diverse Scientific Domains”, the authors present a study of a Evaluating, Ranking, and Clustering (ERC) system anchored in the CCHZ-DISO system. The ERC system is more flexible and simpler than the present system and widely used than CCHZ-DISO system. The innovation of this paper is relatively low. The language in this scientific paper is exaggerated, with a mix of quantitative and qualitative descriptions. The writing is not concise, with completely unnecessary repetitive discussions on the same points.
Specific comments:
- The title should at least mention the “DISO System,” as almost half of the content of the manuscript focuses on this system.
- Line #35-37, please use “any” with caution. The issue of inaccurate wording repeatedly occurs throughout this manuscript.
- Line #69-#73, this paragraph lacks progression in comparison to the previous one.
- Figure 2, 3, 4 look blurry.
- Line #117, How is normalization performed for a single observation?
- Line #147, here the n is the model or countries in this case, it conflicts with former introduction. The “m” represents model in line #114. If it is the case, the sentence should be “where i=1, 2, …, 15”. Usually, in the same manuscript, the meaning of algebraic expressions will remain consistent, especially in nearly identical equations.
- Line #172, j-th “value” is j-th “dimension”.
- Line #173, it is not randomly selection, the selection process needs to be handled with utmost care. For example, after picking a random point from dataset, find second point farthest from the first point, then find third point farthest from the closer of first two points. Pick k points like this and use them as starting cluster centers for the k clusters.
- Line #182, what does “mmm” mean?
- Line #183, K-means is an unsupervised learning to find a local optimum instead of global optimum, it is also a coordinate descent algorithm. Inferring observed values with this method assumes that model values should be nearly identical to observed values with no systematic bias. This approach is equivalent to shooting arrows and then drawing the bullseye where they land.
- Line #423-#427, It is very rare to see such a statement in the discussion section.
technical corrections (typing errors):
- Figure 1, NMSE axis is not consistent with abbreviation used in figure caption.
- Figure 4, Red-green colorblind individuals cannot distinguish the color markers here.
Citation: https://doi.org/10.5194/gmd-2024-82-RC3 -
AC3: 'Reply on RC3', Zengyun Hu, 09 Jul 2024
General comments:
In the manuscript titled “A Unified System for Evaluating, Ranking and Clustering in Diverse Scientific Domains”, the authors present a study of a Evaluating, Ranking, and Clustering (ERC) system anchored in the CCHZ-DISO system. The ERC system is more flexible and simpler than the present system and widely used than CCHZ-DISO system. The innovation of this paper is relatively low. The language in this scientific paper is exaggerated, with a mix of quantitative and qualitative descriptions. The writing is not concise, with completely unnecessary repetitive discussions on the same points.
Reply:
Thanks for your construct suggestions and comments. Some several composite error metrics try to address the overall and comprehensive performance for different models, such as Bergen metrics (Samantaray, et al., 2024), Taylor diagram (Taylor 2001), NSE (Nash and Sutcliffe, 1970) and KGE (Gupta et al., 2008). Our ERC system is not only addressed the overall evaluation quantitatively, but also can solve the ranking and clustering in different scientific domains.
Our innovation and advantages see our reply to Review 2.
Comment:
The title should at least mention the “DISO System,” as almost half of the content of the manuscript focuses on this system.
Reply:
The new title is changed as “ERC: A Unified System for Evaluating, Ranking and Clustering in Diverse Scientific Domains based on DISO”.
Comment:
Line #35-37, please use “any” with caution. The issue of inaccurate wording repeatedly occurs throughout this manuscript.
Reply:
Deleted throughout our manuscript.
Comment:
Line #69-#73, this paragraph lacks progression in comparison to the previous one.
Reply:
Added in the revised manuscript.
Comment:
Figure 2, 3, 4 look blurry.
Reply:
The figure resolution is improved in the revised manuscript.
Comment:
Line #117, How is normalization performed for a single observation?
Reply:
It also can be normalized for a single observation. Referring to our present papers (Hu et al., 2019, 2022). For a specific condition, it has different performation.
Comment:
Line #147, here the n is the model or countries in this case, it conflicts with former introduction. The “m” represents model in line #114. If it is the case, the sentence should be “where i=1, 2, …, 15”. Usually, in the same manuscript, the meaning of algebraic expressions will remain consistent, especially in nearly identical equations.
Reply:
Changed.
Comment:
Line #172, j-th “value” is j-th “dimension”.
Reply:
Changed
Comment:
Line #173, it is not randomly selection, the selection process needs to be handled with utmost care. For example, after picking a random point from dataset, find second point farthest from the first point, then find third point farthest from the closer of first two points. Pick k points like this and use them as starting cluster centers for the k clusters.
Reply:
Changed.
Comment:
Line #182, what does “mmm” mean?
Reply:
It is in Equation (3).
Comment:
Line #183, K-means is an unsupervised learning to find a local optimum instead of global optimum, it is also a coordinate descent algorithm. Inferring observed values with this method assumes that model values should be nearly identical to observed values with no systematic bias. This approach is equivalent to shooting arrows and then drawing the bullseye where they land.
Reply:
Changed.
Comment:
Line #423-#427, It is very rare to see such a statement in the discussion section.
Reply:
Deleted.
Comment:
technical corrections (typing errors):
Figure 1, NMSE axis is not consistent with abbreviation used in figure caption.
Figure 4, Red-green colorblind individuals cannot distinguish the color markers here.
Reply:
Changed.
Citation: https://doi.org/10.5194/gmd-2024-82-AC3
-
RC4: 'Comment on gmd-2024-82', Anonymous Referee #4, 20 Jul 2024
As the previous three reviewers have already done an excellent job pointing out many issues with this paper, I will not repeat their observations. In my opinion, the primary reason for the study's low level of innovation is that the CCHZ-DISO system relies solely on the Euclidean distance to create a composite index when multiple metrics are of interest. While using the Euclidean distance is acceptable, how does it compare with other regularization methods? Additionally, if the CCHZ-DISO system indeed provides a robust metric, where is the mathematical proof of its statistical properties, such as convergence? At the very least, there should be a numerical synthetic test demonstrating how this metric improves modeling practices compared to others.
Citation: https://doi.org/10.5194/gmd-2024-82-RC4 -
AC4: 'Reply on RC4', Zengyun Hu, 21 Jul 2024
The strict mathematical proof of CCHZ-DISO can be found in our first version of CCHZ-DISO (Hu et al., 2019). The advatages of our CCHZ-DISO against other widely applied models/systems have been well illustrated by the third party evaluation, which can be found in the attached file of our response to the Reviewer 2.
Some the third party evaluation are provided as follows.
Kalmar et al. (2021) first recommended the DISO index for assessing historical regional precipitation simulations with the RegCM4.5 model. They compared DISO and Taylor diagram methods and found that “The advantage of using DISO versus the Taylor diagram is that the comprehensive performances of the different models are still not quantified by the latter”.
Deng et al. (2021) introduced DISO, as discussed in section 2.3 of their paper published in the “Journal of Advances in Modeling Earth Systems”. Moreover, they declared that “In this paper, the most important and best advantage of DISO is that after normalizing the observed and simulated data, the value of DISO can express the performance of the same model at different sites”.
Fan et al (2023) recommended that “CCHZ-DISO is a comprehensive evaluation system suitable for multimodels/products, multiple variables, and multiple statistical indicators (Hu et al., 2019, 2022; Zhou et al., 2021). Compared to the widely used Taylor diagram, CCHZ-DISO possesses the advantages of allowing for multivariate comparison and the option to select non-contradictory indicators without the need to satisfy the cosine theorem, which makes it particularly suitable for this paper. The present study constructs a three-dimensional CCHZ-DISO evaluation system by normalizing MB and RMSE into NMB and NRMSE, respectively, and combining them with R,”.
Wu et al (2023) recommended that “Taylor diagrams have been widely used in climate model assessment studies around the world. DISO, also a mathematical statistical model, is an improvement on the Taylor diagram, and therefore, we believe that DISO is suitable for evaluating the simulation capability of climate models around the world. It should be noted, however, that DISO has not yet been applied elsewhere in the world. In a follow-up study, we hope to demonstrate the widespread applicability of DISO.”
Deng, M., Meng, X. and Lu, Y., et al., 2021, Impact and Sensitivity Analysis of Soil Water and Heat Transfer Parameterizations in Community Land Surface Model on the Tibetan Plateau, Journal of Advances in Modeling Earth Systems, 13, e2021MS002670.
Fan, R., Zeng, Z. and Wang, X., et al., 2023, Comprehensive Evaluation and Comparison of AIRS, VASS, and VIRR Water Vapor Products Over Antarctica, Journal of Geophysical Research-Atmospheres, 128
Kalmar, T., Pieczka, H. and Pongracz, R., 2021, A sensitivity analysis of the different setups of the RegCM 4.5 model for the Carpathian region, International Journal of Climatology, 41, E1180-E1201.
Wu, F., Jiao, D. and Yang, X., et al., 2023, Evaluation of NEX-GDDP-CMIP6 in simulation performance and drought capture utility over China-based on DISO, Hydrology Research, 54, 703-721.
I strongly recommend that all the reviewers can read our previous studies (Hu et al., 2019, 2022; Zhou et al., 2021) before they submit their comments.
Hu, Z., Chen, D., Chen, X.*, et al., 2022, CCHZ-DISO: A Timely New Assessment System for data quality or model performance from Da Dao Zhi Jian, Geophysical Research Letters, 49, e2022GL100681.
Zhou, Q., Chen, D., Hu, Z*. and Chen, X*, 2021, Decompositions of Taylor diagram and DISO performance criteria, International Journal of Climatology, 41 (12), 5726-5732.
Hu, Z., Chen, X.*, Zhou, Q., Chen, D., Li, J., 2019, DISO: A rethink of Taylor diagram, International Journal of Climatology, 39(5): 2825-2832.
Citation: https://doi.org/10.5194/gmd-2024-82-AC4
-
AC4: 'Reply on RC4', Zengyun Hu, 21 Jul 2024
Interactive discussion
Status: closed
-
RC1: 'Comment on gmd-2024-82', Anonymous Referee #1, 21 Jun 2024
This study designed a tool for Evaluation, Ranking, and Clustering (ERC) tasks based on Euclidean distance. The main issues are as follows:
- What is the importance and necessity of using a common math mathematical framework to complete Evaluation, Ranking, and Clustering (ERC) tasks from a scientific perspective? The discussion in this research is not sufficient.
- The method established in this paper essentially uses Euclidean distance for evaluation, ranking, and clustering. If there have been any evaluation method based on Euclidean distance for the three fields mentioned above? This study lacks a comprehensive review and summary of existing methods.
- What are the advantages of the method proposed in this study compared to existing methods? What is the framework for evaluating different methods? And how can the superiority of this method be scientifically proven? There is still insufficient work in this study to address the aforementioned issues.
Citation: https://doi.org/10.5194/gmd-2024-82-RC1 -
AC1: 'Reply on RC1', Zengyun Hu, 23 Jun 2024
Comment 1: What is the importance and necessity of using a common math mathematical framework to complete Evaluation, Ranking, and Clustering (ERC) tasks from a scientific perspective? The discussion in this research is not sufficient.
Reply Thanks for your good suggestion. In general, a common mathematical model can be widely applied in numerous scientific domains due to its advantage in revealing the essential of the change law in things. Especially, the common mathematical models usually have the advantages applied in the interdisciplinary areas.
The present models/approaches about Evaluation, Ranking, and Clustering always focus on some special research areas. In essential, they completely can be unified by a common mathematical model since they have the same Euclidean distance characteristics.
Moreover, the ERC system proposed in our study includes some advantages than present Evaluation, Ranking, and Clustering models, which have been illustrated in our manuscript. Therefore, it is very necessary and urgent to unified the present complex and various models about Evaluation, Ranking, and Clustering in a common mathematical model: ERC model/ ERC system.
Comment 2: The method established in this paper essentially uses Euclidean distance for evaluation, ranking, and clustering. If there has been any evaluation method based on Euclidean distance for the three fields mentioned above? This study lacks a comprehensive review and summary of existing methods.
Reply Thanks for your good suggestion. In fact, we also want to search some methods to address the three fields of Evaluation, Ranking, and Clustering. Unfortunately, there is no unified method for the three fields. That is why we propose our ERC system to address the three fields.
Previous studies, such as Taylor diagram (Taylor 2001) Nash-Sutcliffe efficiency (NSE) (Nash and Sutcliffe, 1970) and Kling-Gupta efficiency coefficient (KGE) (Gupta et al., 2008), only employed limited statistical metrics and their applications only focus some special research areas. They are not widely applied in various departments.
References
Gupta, H., Kling, H., Yilmaz, K., and Martinez, G., Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling, Journal of Hydrology, 377, 80-91, doi:10.1016/j.jhydrol.2009.08.003, 2009.
Nash, J.E., Sutcliffe, J.V., River flow forecasting through. Part I. A conceptual models discussion of principles. Journal of Hydrology. 10, 282-290, 1970.
Taylor, K. Summarizing multiple aspects of model performance in a single diagram. Journal of Geophysical Research, 106, 7183-7192, 2001.
Comment 3: What are the advantages of the method proposed in this study compared to existing methods? What is the framework for evaluating different methods? And how can the superiority of this method be scientifically proven? There is still insufficient work in this study to address the aforementioned issues.
Reply Thanks for your good suggestions. The ERC System proposed in this manuscript is our series research of DISO (Distance between Indices of Simulation and Observation) (Hu et al., 2019, 2022; Zhou et al., 2021). ERC not only contains all the advantages of DISO, but also is extended to Ranking, and Clustering. The advantages of ERC System are provided as follows.
1 The dimension of ERC is from one to infinity, which is more flexible and simpler than present system.
2 It can include all the statistical metrics, not some special metrics in other models.
3 It successfully solves the multiple variables, multiple weights in the three fields.
The framework for evaluating different methods is Euclidean distance.
The Third-Party Evaluations of DISO can objective and impartial to show ERC’s advantage as follows.
Since its initial publication, CCHZ-DISO has garnered significant traction and witnessed widespread application, garnering over 100 citations in the Web of Science within the past three years. In this section, we showcase a selection of notable objective and positive evaluations from third-party sources, underscoring the impact and utility of the CCHZ-DISO system.
Kalmar et al. (2021) first recommended the DISO index for assessing historical regional precipitation simulations with the RegCM4.5 model. They compared DISO and Taylor diagram methods and found that “The advantage of using DISO versus the Taylor diagram is that the comprehensive performances of the different models are still not quantified by the latter”. In a sensitivity analysis of soil water and heat transfer parameters in community land surface models, Deng et al. (2021) introduced DISO, as discussed in section 2.3 of their paper published in the “Journal of Advances in Modeling Earth Systems”. Moreover, they declared that “In this paper, the most important and best advantage of DISO is that after normalizing the observed and simulated data, the value of DISO can express the performance of the same model at different sites”.
Wu et al. (2023) suggested that DISO has more advantages than Taylor diagrams, noting that “DISO overcomes some disadvantages of Taylor diagrams and provides an intuitive way to measure differences between various GCMs in the same assessment system”; additionally, the limitations of Taylor diagrams were discussed: “Taylor diagrams (Taylor 2001) are the most common way to assess the performance of climate products. However, they inevitably have inherent drawbacks…. The DISO (Hu et al. 2019) algorithm was designed to overcome the drawbacks that exist in Taylor diagrams. First, DISO has a higher dimensionality than Taylor diagrams… In addition, DISO can evaluate the performance of a model based on multiple metrics at the same time, which can reflect the performance of the model in different aspects”. The comprehensive performance of the CCHZ-DISO has been explored in many studies (Zhuang et al., 2023; Liu et al., 2022; Longo-Minnolo et al., 2022; Ma et al., 2022; Yin et al., 2022).
The third-party evaluations unequivocally indicate that, in comparison to Taylor diagrams, CCHZ-DISO exhibits superior advantages. It stands as an efficient and highly effective approach for the comprehensive quantification of performance across diverse models, providing a holistic assessment of their overall capabilities.
References:
Deng, M., Meng, X. and Lu, Y., et al., 2021, Impact and Sensitivity Analysis of Soil Water and Heat Transfer Parameterizations in Community Land Surface Model on the Tibetan Plateau, Journal of Advances in Modeling Earth Systems, 13, e2021MS002670.
Kalmar, T., Pieczka, H. and Pongracz, R., 2021, A sensitivity analysis of the different setups of the RegCM 4.5 model for the Carpathian region, International Journal of Climatology, 41, E1180-E1201,
Hu, Z., Chen, X. and Zhou, Q., et al., 2019, DISO: A rethink of Taylor diagram. International Journal of Climatology, 39(5), 2825-2832.
Liu, Z., Huang, J. and Xiao, X., et al., 2022, The capability of CMIP6 models on seasonal precipitation extremes over Central Asia, Atmospheric Research, 278, 106364.
Longo-Minnolo, G., Vanella, D. and Consoli, S., et al., 2022, Assessing the use of ERA5-Land reanalysis and spatial interpolation methods for retrieving precipitation estimates at basin scale, Atmospheric Research, 271, 106131.
Ma, R., Xiao, J. and Liang, S., 2022, Pixel-level parameter optimization of a terrestrial biosphere model for improving estimation of carbon fluxes with an efficient model-data fusion method and satellite-derived LAI and GPP data, Geoscientific Model Development, 15, 6637-6657.
Taylor, K., 2001, Summarizing multiple aspects of model performance in a single diagram. Journal of Geophysical Research, 106, 7183-7192.
Citation: https://doi.org/10.5194/gmd-2024-82-AC1
-
RC2: 'Comment on gmd-2024-82', Anonymous Referee #2, 25 Jun 2024
The manuscript proposed an ERC system anchored in the CCHZ-DISO (Chen, Chen, Hu, and Zhou-Distance between Indices of Simulation and Observation) system. Analogous to the CCHZ-DISO system's construction, the ERC system employs the Euclidean distance to perform evaluating, ranking, and clustering tasks. Furthermore, three examples are provided to elucidate the application of the ERC system. In general, the structure of the manuscript is clear, and the expression is smooth. However, the biggest issue with the article is a lack of innovation. Readers will wonder what is truly new and what is particularly special. As one of the principal criteria in peer review, the scientific significance of this manuscript is Poor. It is crucial for the authors to provide a thorough justification for the novelty of their work and to clearly differentiate it from existing research and systems.
- The manuscript claims to present a unified system for evaluating, ranking, and clustering (ERC) based on the CCHZ-DISO system. The novelty would need to be assessed in terms of whether the integration of these three tasks into one system is indeed a new approach or if similar systems have been proposed before.
- The use of Euclidean distance as the mathematical foundation for the ERC tasks is presented as a simple yet potentially powerful method. The innovation here could be in how it is applied across different scientific domains, but it needs to be clear if this application is unique.
- The paper suggests that the ERC system is versatile and can be applied to various scientific fields. The innovation might lie in its adaptability, but it is important to scrutinize whether this cross-domain applicability is truly novel or if other systems have demonstrated similar flexibility. The manuscript posits that the ERC system simplifies complex tasks. Innovation could be in the simplification process itself, but it must be determined if this simplification is indeed novel or if it is a reiteration of existing methods. In section 5, here would be best to have a table comparing existing similar systems, as well as their advantages and disadvantages.
- To assert the novelty of the ERC system, it would be necessary to compare it with existing evaluation, ranking, and clustering methods. The manuscript should clearly articulate why and how the ERC system is different and potentially superior to these methods. In applications in section 6, the evaluation results should include comparisons with other systems. Or, at least, comparisons with other similar assessment systems should be discussed.
- If the manuscript introduces new theoretical concepts or frameworks that underpin the ERC system, these should be clearly outlined and compared with existing theories to highlight their novelty. Similarly, if the innovation lies in the practical application of the system, such as improved efficiency, accuracy, or ease of use, these benefits should be clearly demonstrated and compared with the existing state of the art. The manuscript should provide a clear explanation of why the approach taken is original, including any unique methodologies, algorithms, or data uses that have not been explored in previous research.
- In regard to the applicability of DISO, this paper should especially clarify that the majority of dimensionless measurement indices targeting multiple objectives (or different physical quantities) tend to create a significant amount of ineffective search spaces or paths, thus rendering them less suitable for the field such as the bionic or meta-heuristic artificial intelligence algorithms.
- The authors mentioned the flexibility in selecting statistical metrics for the ERC system. However, it would be helpful to provide guidance on the criteria for selecting these metrics in the context of geoscientific applications, where data characteristics can be quite varied.
- The discussion on the comparison between NSE, KGE, and CCHZ-DISO is appreciated. However, it would be advantageous to include a more comprehensive comparison with other prevalent methods in geosciences, such as the Taylor diagram (and does the DISO’s advantage is simpler?), to position the ERC system within the field better.
- The paper briefly touches on the significance testing for models with small differences in ERC values. It would be valuable to see a more detailed explanation of how this testing is conducted and its implications for geoscientific studies, where subtle differences can be critical.
- The authors state that the ERC system does not consider data characteristics such as outliers. Given the common presence of outliers in geoscience data, it is important to discuss how the ERC system's results might be affected and whether any adjustments or preprocessing are recommended.
- Technical corrections, i.e.,
In Line 340, repeated “clustering”
Figure 3, the position of its reference to is inaccurate, and all abbreviations of models should be given their full names, as well as brief introductions of different models, for broader reading.
Citation: https://doi.org/10.5194/gmd-2024-82-RC2 - AC2: 'Reply on RC2', Zengyun Hu, 03 Jul 2024
-
RC3: 'Comment on gmd-2024-82', Anonymous Referee #3, 08 Jul 2024
General comments:
In the manuscript titled “A Unified System for Evaluating, Ranking and Clustering in Diverse Scientific Domains”, the authors present a study of a Evaluating, Ranking, and Clustering (ERC) system anchored in the CCHZ-DISO system. The ERC system is more flexible and simpler than the present system and widely used than CCHZ-DISO system. The innovation of this paper is relatively low. The language in this scientific paper is exaggerated, with a mix of quantitative and qualitative descriptions. The writing is not concise, with completely unnecessary repetitive discussions on the same points.
Specific comments:
- The title should at least mention the “DISO System,” as almost half of the content of the manuscript focuses on this system.
- Line #35-37, please use “any” with caution. The issue of inaccurate wording repeatedly occurs throughout this manuscript.
- Line #69-#73, this paragraph lacks progression in comparison to the previous one.
- Figure 2, 3, 4 look blurry.
- Line #117, How is normalization performed for a single observation?
- Line #147, here the n is the model or countries in this case, it conflicts with former introduction. The “m” represents model in line #114. If it is the case, the sentence should be “where i=1, 2, …, 15”. Usually, in the same manuscript, the meaning of algebraic expressions will remain consistent, especially in nearly identical equations.
- Line #172, j-th “value” is j-th “dimension”.
- Line #173, it is not randomly selection, the selection process needs to be handled with utmost care. For example, after picking a random point from dataset, find second point farthest from the first point, then find third point farthest from the closer of first two points. Pick k points like this and use them as starting cluster centers for the k clusters.
- Line #182, what does “mmm” mean?
- Line #183, K-means is an unsupervised learning to find a local optimum instead of global optimum, it is also a coordinate descent algorithm. Inferring observed values with this method assumes that model values should be nearly identical to observed values with no systematic bias. This approach is equivalent to shooting arrows and then drawing the bullseye where they land.
- Line #423-#427, It is very rare to see such a statement in the discussion section.
technical corrections (typing errors):
- Figure 1, NMSE axis is not consistent with abbreviation used in figure caption.
- Figure 4, Red-green colorblind individuals cannot distinguish the color markers here.
Citation: https://doi.org/10.5194/gmd-2024-82-RC3 -
AC3: 'Reply on RC3', Zengyun Hu, 09 Jul 2024
General comments:
In the manuscript titled “A Unified System for Evaluating, Ranking and Clustering in Diverse Scientific Domains”, the authors present a study of a Evaluating, Ranking, and Clustering (ERC) system anchored in the CCHZ-DISO system. The ERC system is more flexible and simpler than the present system and widely used than CCHZ-DISO system. The innovation of this paper is relatively low. The language in this scientific paper is exaggerated, with a mix of quantitative and qualitative descriptions. The writing is not concise, with completely unnecessary repetitive discussions on the same points.
Reply:
Thanks for your construct suggestions and comments. Some several composite error metrics try to address the overall and comprehensive performance for different models, such as Bergen metrics (Samantaray, et al., 2024), Taylor diagram (Taylor 2001), NSE (Nash and Sutcliffe, 1970) and KGE (Gupta et al., 2008). Our ERC system is not only addressed the overall evaluation quantitatively, but also can solve the ranking and clustering in different scientific domains.
Our innovation and advantages see our reply to Review 2.
Comment:
The title should at least mention the “DISO System,” as almost half of the content of the manuscript focuses on this system.
Reply:
The new title is changed as “ERC: A Unified System for Evaluating, Ranking and Clustering in Diverse Scientific Domains based on DISO”.
Comment:
Line #35-37, please use “any” with caution. The issue of inaccurate wording repeatedly occurs throughout this manuscript.
Reply:
Deleted throughout our manuscript.
Comment:
Line #69-#73, this paragraph lacks progression in comparison to the previous one.
Reply:
Added in the revised manuscript.
Comment:
Figure 2, 3, 4 look blurry.
Reply:
The figure resolution is improved in the revised manuscript.
Comment:
Line #117, How is normalization performed for a single observation?
Reply:
It also can be normalized for a single observation. Referring to our present papers (Hu et al., 2019, 2022). For a specific condition, it has different performation.
Comment:
Line #147, here the n is the model or countries in this case, it conflicts with former introduction. The “m” represents model in line #114. If it is the case, the sentence should be “where i=1, 2, …, 15”. Usually, in the same manuscript, the meaning of algebraic expressions will remain consistent, especially in nearly identical equations.
Reply:
Changed.
Comment:
Line #172, j-th “value” is j-th “dimension”.
Reply:
Changed
Comment:
Line #173, it is not randomly selection, the selection process needs to be handled with utmost care. For example, after picking a random point from dataset, find second point farthest from the first point, then find third point farthest from the closer of first two points. Pick k points like this and use them as starting cluster centers for the k clusters.
Reply:
Changed.
Comment:
Line #182, what does “mmm” mean?
Reply:
It is in Equation (3).
Comment:
Line #183, K-means is an unsupervised learning to find a local optimum instead of global optimum, it is also a coordinate descent algorithm. Inferring observed values with this method assumes that model values should be nearly identical to observed values with no systematic bias. This approach is equivalent to shooting arrows and then drawing the bullseye where they land.
Reply:
Changed.
Comment:
Line #423-#427, It is very rare to see such a statement in the discussion section.
Reply:
Deleted.
Comment:
technical corrections (typing errors):
Figure 1, NMSE axis is not consistent with abbreviation used in figure caption.
Figure 4, Red-green colorblind individuals cannot distinguish the color markers here.
Reply:
Changed.
Citation: https://doi.org/10.5194/gmd-2024-82-AC3
-
RC4: 'Comment on gmd-2024-82', Anonymous Referee #4, 20 Jul 2024
As the previous three reviewers have already done an excellent job pointing out many issues with this paper, I will not repeat their observations. In my opinion, the primary reason for the study's low level of innovation is that the CCHZ-DISO system relies solely on the Euclidean distance to create a composite index when multiple metrics are of interest. While using the Euclidean distance is acceptable, how does it compare with other regularization methods? Additionally, if the CCHZ-DISO system indeed provides a robust metric, where is the mathematical proof of its statistical properties, such as convergence? At the very least, there should be a numerical synthetic test demonstrating how this metric improves modeling practices compared to others.
Citation: https://doi.org/10.5194/gmd-2024-82-RC4 -
AC4: 'Reply on RC4', Zengyun Hu, 21 Jul 2024
The strict mathematical proof of CCHZ-DISO can be found in our first version of CCHZ-DISO (Hu et al., 2019). The advatages of our CCHZ-DISO against other widely applied models/systems have been well illustrated by the third party evaluation, which can be found in the attached file of our response to the Reviewer 2.
Some the third party evaluation are provided as follows.
Kalmar et al. (2021) first recommended the DISO index for assessing historical regional precipitation simulations with the RegCM4.5 model. They compared DISO and Taylor diagram methods and found that “The advantage of using DISO versus the Taylor diagram is that the comprehensive performances of the different models are still not quantified by the latter”.
Deng et al. (2021) introduced DISO, as discussed in section 2.3 of their paper published in the “Journal of Advances in Modeling Earth Systems”. Moreover, they declared that “In this paper, the most important and best advantage of DISO is that after normalizing the observed and simulated data, the value of DISO can express the performance of the same model at different sites”.
Fan et al (2023) recommended that “CCHZ-DISO is a comprehensive evaluation system suitable for multimodels/products, multiple variables, and multiple statistical indicators (Hu et al., 2019, 2022; Zhou et al., 2021). Compared to the widely used Taylor diagram, CCHZ-DISO possesses the advantages of allowing for multivariate comparison and the option to select non-contradictory indicators without the need to satisfy the cosine theorem, which makes it particularly suitable for this paper. The present study constructs a three-dimensional CCHZ-DISO evaluation system by normalizing MB and RMSE into NMB and NRMSE, respectively, and combining them with R,”.
Wu et al (2023) recommended that “Taylor diagrams have been widely used in climate model assessment studies around the world. DISO, also a mathematical statistical model, is an improvement on the Taylor diagram, and therefore, we believe that DISO is suitable for evaluating the simulation capability of climate models around the world. It should be noted, however, that DISO has not yet been applied elsewhere in the world. In a follow-up study, we hope to demonstrate the widespread applicability of DISO.”
Deng, M., Meng, X. and Lu, Y., et al., 2021, Impact and Sensitivity Analysis of Soil Water and Heat Transfer Parameterizations in Community Land Surface Model on the Tibetan Plateau, Journal of Advances in Modeling Earth Systems, 13, e2021MS002670.
Fan, R., Zeng, Z. and Wang, X., et al., 2023, Comprehensive Evaluation and Comparison of AIRS, VASS, and VIRR Water Vapor Products Over Antarctica, Journal of Geophysical Research-Atmospheres, 128
Kalmar, T., Pieczka, H. and Pongracz, R., 2021, A sensitivity analysis of the different setups of the RegCM 4.5 model for the Carpathian region, International Journal of Climatology, 41, E1180-E1201.
Wu, F., Jiao, D. and Yang, X., et al., 2023, Evaluation of NEX-GDDP-CMIP6 in simulation performance and drought capture utility over China-based on DISO, Hydrology Research, 54, 703-721.
I strongly recommend that all the reviewers can read our previous studies (Hu et al., 2019, 2022; Zhou et al., 2021) before they submit their comments.
Hu, Z., Chen, D., Chen, X.*, et al., 2022, CCHZ-DISO: A Timely New Assessment System for data quality or model performance from Da Dao Zhi Jian, Geophysical Research Letters, 49, e2022GL100681.
Zhou, Q., Chen, D., Hu, Z*. and Chen, X*, 2021, Decompositions of Taylor diagram and DISO performance criteria, International Journal of Climatology, 41 (12), 5726-5732.
Hu, Z., Chen, X.*, Zhou, Q., Chen, D., Li, J., 2019, DISO: A rethink of Taylor diagram, International Journal of Climatology, 39(5): 2825-2832.
Citation: https://doi.org/10.5194/gmd-2024-82-AC4
-
AC4: 'Reply on RC4', Zengyun Hu, 21 Jul 2024
Viewed
| HTML | XML | Total | Supplement | BibTeX | EndNote | |
|---|---|---|---|---|---|---|
| 401 | 65 | 39 | 505 | 43 | 15 | 17 |
- HTML: 401
- PDF: 65
- XML: 39
- Total: 505
- Supplement: 43
- BibTeX: 15
- EndNote: 17
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
Zengyun Hu
Deliang Chen
Zhuo Zhang
Qiming Zhou
Qingxiang Li
This preprint has been withdrawn.
- Preprint
(2242 KB) - Metadata XML
-
Supplement
(126 KB) - BibTeX
- EndNote