The submitted paper discusses an interesting, novel and computationally efficient approach to model evaluation which builds on concepts of ensemble Kalman filtering and analog approaches to build skill scores which indicate some degree of skill in identifying model errors when assessed in a perfect model framework. The technique is demonstrated for a highly idealised case (the Lorenz 63 model) and an intermediate complexity climate model.



The paper is well written, and novel.  My opinion is that it should be published with only minor edits.



Minor comments:



1 The approach described here is acceptable as a proof of concept - however it is likely not an optimal use of the data used in the training simulation used to assemble the analogs.  In particular, the use of only small-scale information in the construction of analogs is discarding valuable information which would be represented in the covariance structure of the model output.  The need to minimise the state space of model in order to find acceptable analogs is clear - but my suspicion is that a compression of state space which preserves elements of large scale covariance (such as PCA), rather than isolated regional analyses, would be even more effective.



2. A discussion of the dependency of performance on training run length for the analog would be useful, compared to a forecast-based approach (in the Lorenz case), and in terms of the ability to distinguish model errors (for SPEEDY).



4. Though the authors have demonstrated that CME provides a generally improved regional assessment of model error, this is not universally the case - especially for RH08, where ME provides a stronger signal in a number of regions.  A short discussion on regions where this occurs, and potentially why, would be useful.