In their manuscript,  Lee et al. present the  PCDMI Metrics Package (PMP) version 3, which is an open-source Python software package that provides tools for comparisons of ESMs with each other as well as  for comparisons of ESMs with observations. A large range of atmospheric processes can be assessed with this software package. Since this package has been developed within the CMIP comparison projects the PMP results can be produced for all model simulations contributing to CMIP6 and earlier CMIP phases.  

The manuscript is generally well written and the package useful for the scientific community, but I have several comments that should be taken into account before publication in GMD. Generally, the authors should take care that their message comes better through. At the moment, I have the feeling that there is a lot written, but what is the take home message? Is this a unique package or are there other, similar packages available. What is new or unique for your specific software package?

General comments:

• This is a quite extensive overview and I was wondering how a new user should get started with PMP. I saw from the links you provided that everything is quite well documented, however, this is so much information at once that I think that a new user will have really trouble getting started. Is there e.g. a documentation that could be downloaded as a single pdf file or does one really spend hours reading everything on the computer screen?
• How can a user install PMP on a computer? What requirements are needed? You provide a link to your github repository, where this information is provided, but I think this information should also be provided in the manuscript itself.
• A general questions I was wondering about is, if you are offering workshops for training of new users?
• In the abstract you state what you will discuss the history up to date, recent updates and future directions. The future directions are discussed to some part in Section 6, but I could not find any information or discussion on the history of PMP up to date or what the updates between former and this version were.
• Generally, all sections seem to be a bit too lengthy and to my opinion not really coming to the point. Best example is the summary and future directions section. Although you have a discussion section, you provide a quite long summary and future directions section without really summarizing what you have presented.
• Is this the only tool for analyzing CMIP data or have there also other tools been developed?

Specific comments:

P2, L47-48: When was version 1 developed (published)? What are the major changes/new developments you are presenting here?

P3, L82: Also here you should clearly state when the first version was developed. Has this version been somewhere documented/published? Or has this version just been provided to the CMIP community?

P4, L86: Here, some examples should be given. Which performance metrics or statistical measures are used?

P4, L101: Also here, add some examples.

P4, L102: Why only a subset? Specify.

P4, L102: What do you mean with CMIP class models? Models that participate in CMIP or the kind of models participating in CMIP?

P4, L109: References? Are there any publications?

P6, L170: The following subsections are rather the “processes” that can be assessed with PMP than metrics. The statistics you are using are the metrics for each of this processes.

P6, L174: “well-established statistics” -> You should more clearly write here which statistics are used (at least some examples should be given).

P6, L178: Provide here a typical example. What is a typical default model and what are the alternatives.

P6, L189-191: Provide a short explanation how this is done or where this is described.

P8, L237: Since “Performance”, “Processes” and “Teleconnections” are not really metrics, I would suggest to rewrite the sentence as follows: “The ENSO metrics used to assess/evaluate the models are divided into three categories: ………..”.

P16-18: Summary section is too lengthy and not really summarizing what has been presented in the manuscript. Some part of this should rather be part of the discussion section. Further, there should be a conclusion section e.g. stating clearly what is the gain for the community of this software package. Has it already successful applied  for CMIP etc.

Figure 1 caption: Add in the figure caption what is shown in the boxes (thus, add that the separation of RMSE by season is shown there)

Technical corrections:

P4, L108: Add “assessment” or “model” after CMIP.

P8, L247: Abbreviation “ITCZ” has not been introduced.

P9, L288: Abbreviation ”DOE” has not been introduced.

P11, L344: Abbreviation GoG has not been  introduced. It is given in the figure caption

of figure 7, but not in the text at L338: Should be done there, too.

P12, L401: Although the abbreviations “WGNE” and “WGCM” have been introduced, I would suggest to repeat it here.

P12, L 466: Abbreviation “ESGF” has not been introduced.

P15, L493: …….Section 3.3 are respectively -> Section 3.3., respectively, are

P16, L542: use parenthesis instead of brackets

P18, L605: check sentence.

P41, Figure 6 caption, L1293: units should be given in an upright font.

COMMENTS TO THE AUTHOR(S)

A comprehensive and objective evaluation of Earth System Models (ESMs) is essential to understand the strengths and weaknesses of individual ESMs and to provide a basis for model improvement. This paper provides an overview of the PCMDI Metric Package (PMP) Version 3, which provides a "quick-look" objective comparison of ESMs to each other and to observations. The purpose, flow, and explanation of the present paper were well written in a concise and easy to understand manner. I expect that the PMP package will be more useful in the context of the CMIP7 project and subsequent phases. I have only a few suggestions before the publication of this paper in the GMD journal.

General suggestion:

I am so curious about how to install the PMP package on my local computer as a user, and what are the requirements for a successful installation. This is the most basic and critical question that readers may have. While the author provides a GitHub link for the related information, I believe it is essential to have a dedicated section in this paper with a brief explanation. This will serve as a catalyst for more readers to become actual PMP users.

Minor comments:

Lines 101-102: I am just wondering if the PMP can be used to evaluate the regional climate models that participated in the RCM project, such as CORDEX. The authors could briefly discuss this or any related future plans (if the authors have any) in the discussion section.

Figure 1. Are all model grids (or just land?) used in the evaluation? It would be helpful to indicate this in the figure caption.

For Abbreviation:

Line 37 (Abstract): Provide the PCMDI's full name.

Line 247. Provide the ITCZ's full name.

Line 288. Provide the DOE's full name.

Line 344. Provide the GoG's full name.

Line 466. Provide the ESGF's full name.