Dear T. Hallouin and co-authors,

I really enjoyed reading this article, and evalhyd seems a very nice verification too: felicitations! I have uploaded a file with some minor suggestions, while here with this following comment I wish to bring the attention of the scientific community to one aspect of verification, for the general online discussion. 

I was particularly triggered about the option in evalhyd of performing conditional verification. I will share here some of our own experience (at the Canadian Met Service) with conditional verification, which maybe can inspire further developments in the tool and, more in general, awareness in the interpretation of the results.

Conditioning on the verification sample can have strong impacts on the verification results (e.g. it can flip the sign of a bias), and hence allows in-depth analysis and understanding of the prediction performance, since the conditioning is usually related to physically-driven phenomena. In a sense, conditional verification is the first step towards process-based diagnostics.

In verification exercises which include several variables (e.g. pressure, temperature, clouds, etc) applying a condition to a variable while verifying a different variable is the common practice (as an example, verification of surface temperature in cloudy versus clear-sky conditions inform of the model performance in reproducing the radiation budget). The condition, however, should be applied to both observed and forecast values (e.g. forecast AND observation being cloudy): I will refer to this double condition as bilateral. On the other hand, when a unilateral condition is applied, to only the observed or forecast variable (e.g. cloudy conditions only for the forecast) this can synthetically introduce a bias in the verification results: in the cloud/temperature example above, stratifying for cloudy conditions only in the forecast leads to a synthetic warm bias for the surface temperature, because in the sample there are bound to be both cloudy and clear sky observations, and when the observations have clear sky the surface temperature is expected to be colder. In other words, the bilateral condition will sample all the “hits” for cloudy sky, whereas the unilateral condition will sample the “hits” and “false alarms” for cloudy sky. From our experience, we advise bilateral over unilateral conditioning. (Of course one can also do unilateral conditioning, but need to be aware of the introduced biases in the interpretation of the verification results).

Applying the unilateral condition to the same variable which is verified might also lead to synthetic biases. As an example, if you stratify your sample for the strong predicted stream flows, you are bound to include in the sample several strong observed stream flows (the “hits”), but also some average or weak observed stream-flows (because the prediction might have some “false alarms”). Then you tend to “artificially” diagnose over-prediction for the strong stream flow (and vice-versa for the low stream flow, conditioning only on the prediction you are bound to find under-estimation, because in your sample you’ll have some observed events which are medium or strong).

I was amazed (also a bit puzzled) to see that in your Figure 6 you have opposite results than I expected (underprediction for high predicted stream flow, more overprediction for low predicted stream flows; the under-dispersion for the average predicted stream flow is instead expected). For me it would be interesting to understand why, is it due to the characteristics of streamflow prediction (where the timing is always predicted well, and hence false alarms and misses are very rare)? what is the behaviour in the other stations? What would you obtain with the bilateral condition?

I would be grateful if you could add in the article some discussion about unilateral versus bilateral conditions.

Thank you + Bests                  Barbara Casati

The paper introduces evalhyd, an interesting software tool designed for the evaluation of streamflow predictions. The tool's commitment to standardization and open-source accessibility is commendable, providing a valuable contribution to enhancing reproducibility in hydrological studies. Notably, the well-thought-out design principles, which incorporate a compiled C++ core and thin bindings for multiple languages, contribute to the tool's efficiency and usability. 

However, despite the paper positioning evalhyd as a contribution to hydroinformatics, the manuscript's focus on the technical aspects of model development limits its scientific impact. The paper could benefit from a more explicit emphasis on the broader scientific implications and advancements in hydrologic science that the tool facilitates. Some specific comments are as follows:

The introduction would benefit from more clearly articulating what new capabilities evalhyd provides compared to existing hydrologic evaluation packages. As it stands, the motivation around standardization across languages is a bit weak.

In the key functionalities section, the masking and bootstrapping methods need more detailed explanation. Pseudocode or formulas would help make these clearer. For the evaluation metrics, links or references to the original sources for each metric should be provided. More justification for the specific metrics included would also help show the comprehensiveness.

The case study,  more novel demonstrations of the tool would strengthen this section.

The conclusions would be improved by specifically emphasizing the limitations around extensibility, visualizations, and support for continuous distributions. Comparisons to other existing packages may help contextualize the pros/cons.

Overall more critical analysis is needed on how evalhyd improves on the current state of the art in hydrologic evaluation tools. The paper currently lacks motivation and innovation.