Review report attached as a PDF file.

In the paper, the authors introduce ICONEval, an open-source evaluation framework built on top of the Earth System Model Evaluation Tool (ESMValTool) designed for rapid, reproducible, and automated performance monitoring during the development of Earth System Models (ESMs). ICONEval introduces parallel processing and HTML summary reports categorized into three diagnostic pillars: sanity checks against observational bounds, fundamental physical consistency checks (e.g., mass conservation), and multi-domain climate fidelity diagnostics. The tool is demonstrated using the ICON-XPP model and is positioned as a highly efficient benchmarking utility for comparing model configurations, testing updated parameterizations, and assessing emerging hybrid machine learning-enhanced climate models.

This manuscript is well-suited for the journal's scope, as it introduces an open-source evaluation framework (ICONEval) that directly advances our ability to benchmark, monitor, and improve ESMs, including emerging hybrid machine learning configurations. The paper is well-prepared, presenting a highly structured and logical three-pillar approach—spanning sanity checks, physical consistency, and climate fidelity—that is clearly articulated and thoroughly demonstrated using the ICON-XPP model. Given the increasing complexity of modern climate modeling, the development of reproducible, automated, and customized tools for specific models is highly relevant to the journal’s audience and represents a timely, high-quality contribution to the community.

I suggest accepting the manuscript subject to the authors addressing the following a few specific comments.

Specific comments:

Line 145 and Figure 2: Still, it is not very clear to me that how the “reasonable” upper and lower limits (red lines in the figure) were defined, in particular with using multiple reference datasets. For example, for minimum, was the minimum monthly value across all years of global spatial average taken from each reference dataset, then minimum value among them was taken as the “reasonable minimum lower limit?” I suggest further clarifying it in the manuscript.

Figure 8a: While I understand there could be some artifact due to difference in the data frequency interval (i.e., ERA5 1 hourly while ICON-XPP 6 hourly), the sudden magnitude change over the pacific seems odd, which could have been resulted from incorrect local time handling. Could you please double check on this?

Figure 9. In addition to showing the difference field only, I suggest showing the full field from each data (ICON-XPP and ERA5) as well, so the pattern could be easily compared, like in Figure 10.

Figure 11. In the Historical experiment, year-to-year comparison between model and observation does not say much, as their matching is not even expected. Although the authors have described this in the manuscript (line 275-276), I wonder if it would be beneficial to re-emphasize this point in the figure caption so advise readers what to focus on the figure. One additional option could be comparing the standard deviation or variance values obtained from each time series as an indicator for the intensity of the variability, or trend line. The same comment applies to Figs. 12 and 14 as well. For Fig. 11, I am not sure how much useful information could be derived from the CMIP6 results in the background.