I have had the opportunity to review the paper entitled "The Ensemble Consistency Test: From CESM to MPAS and beyond" by Price-Broncucia et al. 

I think it is a very good paper which deserves publication at GMD. Initially, at a first glance to the introduction, it seemed to me that it was more oriented towards technical aspects of model technical testing. However, as the paper described the methodology that the authors use to identify potential errors, the way they take into account the expected numbers of False Alarms and so on, I realized that the paper went clearly (in my humble opinion) beyond the point of a "technical contribution for model testing" to the next level. The authors have successfully described the way an existing methodology can be extended to new models. And the authors make the tool available to other model developers. 

As such, I support the paper being accepted, as I really feel it can be a valid contribution for other members of the community. I was shocked by the results in section 5.3, the system automatically detecting these errors with such a low number of model steps is a very good result, from my point of view.

However, there are some results or ways that the authors use to solve the problem which puzzle me. I think that the paper would be better self-contained if the authors discussed the following points. I guess that the authors have some reasons to use the technique the way they use it. But I have my own (may be wrong) ideas, questions or, basically constructive suggestions, which I am sure the authors may answer easily. 

1. I am surprised by the way the authors use the PCs by averaging out the variables over the whole domain. It is true that PCs are good for isolating the main axes of variability in a dataset. However, it is also clear that there are some solid spatial structures in atmospheric fields that are rooted in basic physics. I am thinking myself in terms of the latitudinal structure of zonally averaged surface temperature. Or I can also consider the vertical temperature profile of the middle troposphere or the stratosphere. These features are missed when the variables are averaged over the full domain before calculating the PCs. I am also aware of the fact that the way these features are represented might be very dependent on the horizontal/vertical resolution of the model being tested. However, checking the zonally averaged surface temperature in midlatitudes ([30,60] degrees in every hemisphere) and tropical temperatures [20S,20N] before calculating the PCs might give three different series which would be representing (even in a crude way) the meridional temperature gradient. Thinking in the same way, getting the temperature at 400 hPa minus the temperature at 700 hPa seems a crude but simple way of evaluating the vertical profile in the troposphere. A similar technique might be used to evaluate the lapse rate around 100 hPa, for instance, adding two different columns to the PCs (without removing the averages that the authors use). Using these diagnostics might improve the sensitivity of the system, can be extended to models of different resolutions and does not involve significant new computations. What do the authors think about this? Can they elaborate on this? 
2. Line 293. The authors state that a correlation coefficient of 0.75 is a “limit” that they use to diagnose whether a correlation matrix is/is not rank deficient. However, this would be strongly dependent on the number of degrees of freedom. Why do not they use the spectrum of singular values derived from the singular value decomposition (SVD) of the correlation matrix? If no singular value is negligible, they know for sure they don’t have a problem. Since they apply this a “small” matrix of averaged series, using the SVD shouldn’t be computationally very expensive. 
3. Line 332. The authors use the (often used) target of 95% of explained variance to identify the number of PCs that must be kept. I don’t think this is critical, but I suggest the authors (for future versions of the software) that determining whether the corresponding EOFs are or are not well determined by the sample might be safer from the point of view of the stability of the next steps. There are alternative methods in the literature for this, either based on the errors of the eigenvalues or the congruence coefficients. 
   1. Cheng, X., G. Nitsche, and J. M. Wallace, 1995: Robustness of low-frequency circulation patterns derived from EOF and rotated EOF analyses. J. Climate, 8, 1709–1713. 
   2. North, G. R., T. L. Bell, R. F. Cahalan, and F. J. Moeng, 1982: Sampling errors in the estimation of empirical orthogonal functions. Mon. Wea. Rev., 110, 699–706.
4. I think the sentence in lines 174 and 175 must be better worded, I don’t find it clear enough. 

Besides those comments, there are still a few typos in the paper. My mother tongue is not English, but I guess the authors should still make a thorough reading of the text. 

1. Line 174. “the the”
2. Line 314. Is a “d” missing after “average”?

3. Line 454. “with with”