the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
The E3SM Diagnostics Package (E3SM Diags v2.6): A Python-based Diagnostics Package for Earth System Models Evaluation
- 1Lawrence Livermore National Laboratory, Livermore, CA, USA
- 2Los Alamos National Laboratory, Los Alamos, NM, USA
- 3University of California, Irvine, Irvine, CA, USA
- 4Brookhaven National Laboratory, Upton, NY, USA
- 5National Center for Atmospheric Research, Boulder, CO, USA
- 6Oak Ridge National Laboratory, Oak Ridge, TN, USA
- 7Pacific Northwest National Laboratory, Richland, WA, USA
- 8University of California, Los Angeles, Los Angeles, CA, USA
- 9University of California, Davis, Davis, CA, USA
- anow at: Google LLC, Mountain View, CA, USA
- 1Lawrence Livermore National Laboratory, Livermore, CA, USA
- 2Los Alamos National Laboratory, Los Alamos, NM, USA
- 3University of California, Irvine, Irvine, CA, USA
- 4Brookhaven National Laboratory, Upton, NY, USA
- 5National Center for Atmospheric Research, Boulder, CO, USA
- 6Oak Ridge National Laboratory, Oak Ridge, TN, USA
- 7Pacific Northwest National Laboratory, Richland, WA, USA
- 8University of California, Los Angeles, Los Angeles, CA, USA
- 9University of California, Davis, Davis, CA, USA
- anow at: Google LLC, Mountain View, CA, USA
Abstract. The E3SM Diagnostics Package (E3SM Diags) is a modern, Python-based Earth System Model (ESM) evaluation tool (with Python module name e3sm_diags), developed to support the Department of Energy (DOE) Energy Exascale Earth System Model (E3SM). E3SM Diags provides a wide suite of tools for evaluating native E3SM output, as well as ESM data on regular latitude longitude grids, including output from Coupled Model Intercomparison Project (CMIP) class models.
E3SM Diags is modeled after the National Center for Atmospheric Research (NCAR) atmospheric model working group (AMWG) diagnostics package. In its version 1 release, E3SM Diags included a set of core essential diagnostics to evaluate the mean physical climate from model simulations. As of version 2.6, more process-oriented and phenomenon-based evaluation diagnostics have been implemented, such as analysis of the Quasi-biennial Oscillation (QBO), El Niño – Southern Oscillation (ENSO), streamflow, diurnal cycle of precipitation, tropical cyclones and ozone. An in-situ dataset from DOE’s Atmospheric Radiation Measurement (ARM) program has been integrated into the package for evaluating the representation of simulated cloud and precipitation processes.
This tool is designed with enough flexibility to allow for the addition of new observational datasets and new diagnostic algorithms. Additional features include: customizable figures; streamlined installation, configuration and execution; and multiprocessing for fast computation. The package uses an up-to-date observational data repository maintained by its developers, where recent datasets are added to the repository as they become available. Finally, several applications for the E3SM Diags module were introduced to fit a diverse set of use cases from the scientific community.
Chengzhu Zhang et al.
Status: closed
-
RC1: 'RC Comment on gmd-2022-38', Valeriu Predoi, 10 May 2022
Review also available at https://github.com/valeriupredoi/review_gmd-2022-38/blob/master/review_gmd-2022-38.md
Paper title: The E3SM Diagnostics Package (E3SM Diags v2.6): A Python-basedDiagnostics Package for Earth System Models Evaluation
Preprint: https://gmd.copernicus.org/preprints/gmd-2022-38/gmd-2022-38.pdfReview data
- Review by: Valeriu Predoi
- Date: 10 May 2022General remarks
The paper reads well, and contains about the right balance between technical vs scientific aspects, however, I have a couple suggestions:
- is this a technical overview work or a more scientifically-oriented description of the diags/metrics in E3SM_Diags? It'd be worth stating this clearly at the very top;
- the technical aspect opens the narrative, with a general description of the package's workflow, only to be interrupted by a very detailed scientific description of a number of standard metrics, and, after these metrics have been explored, the narrative comes back to technical aspects related to installation, I/O etc. It would be good to have the technical aspects bunched together, to allow the reader to follow through, and understand how the thing works; same for the metrics/diagnostics - probably better understood after the reader has insight into the compute workflow aspects of E3SM_Diags;
- versions matter: what is the purpose of describing E3SM_Diags v2 (actually, a very specific v2.6) - are there any major enhancements/refactorings/performance improvements from v1? Is there a paper describing v1, and if so, please link it? If it was me, I'd really drop the very specific v2.6 and instead refer to the tool with a more general v2 only;
- probably avoid including very specific script filenames (e.g. `postprocessing_E3SM_data_for_TC_analysis.sh`), these are documented in the code documentation already; the reader should get a bird's eye of the workflow without getting them into the nitty-gritty details;
- some information about the code license is needed (GPL, Apache etc)General suggestions
- title: E3SM Diags v2.6 is mentioned, but 2.6.1 on conda-forge https://anaconda.org/conda-forge/e3sm_diags
suggestion: either change to "2.6+" or don't specify the version in the title, or specify why 2.6 is a major
release even though the number doesn't suggest that way;
- Abstract: please provide a reference/link for AMWG diagnostics package;
- Introduction: line 27: "Most tools listed here are focused on the atmosphere" - not entirely true, ESMValTool
has a solid component for ocean model data evaluation; suggest changing to "Most tools listed here
have started with a main focus on the atmosphere, but evolved to analyze other ESM realms too"
- Introduction: line 32 - please provide a reference (methods paper, documentation and/or software repository) for AMWG
- Introduction: line 57 - "...project takes a distinct but related approach" - contradiction in terms
- Introduction: line 63 "Distribution of these Python packages is now mostly accomplished through Conda" - please provide
a reference to the Anaconda documentation/main page; also note that "conda" is the executable and the
package manager tool, but Anaconda is the organization and the actual package manager, so it's probably best to
use Anaconda in this context, maybe even "Anaconda/Miniconda"
- Introduction: line 72 - "This paper is a comprehensive description of E3SM Diags (as of version 2.6)" - anything special
about v2.6? Please briefly describe why the choice of version;
- Table 1: I am assuming the references will be available as active links in the final version of the paper?
(idem): ESMValTool: AR5 and AR6, please, also - ESMValTool is a PyPi package too https://pypi.org/project/ESMValTool/
- Optional: ESMValTool handles OBS datasets too, that are not necessarily CMIP-like file formats, via a process
of CMOR-standardization (CMORization), maybe with mentioning?
- Tech overview: line 76 - maybe provide a link to the installation section of your documentation?
- Tech overview: line 83: multi-threading, really, up to you if you want to change from multiprocessing to multi-threading
- Tech overview: general: it would be nice to provide links to various main sections of the package documentation
e.g. when talking about installation, configuration, running instructions
- S3.1: line 128: how is the reference grid chosen?
- S3.2: line 157: a very detailed explanation on how the (calibration) run works, but a rather frugal explanation of how CMIP data is retrieved from an ESGF node: "The built-in derived variable module takes in CMIP variables and handles variable name and unit conversions" - it'd be good to have this explained a bit better, and in a user-friendly way e.g. data specifications include: var name, mip, experiment name, frequency etc - with these parameters gathered from user input, the code goes and grabs data from the ESGF via standard-path DRS structure conventions etc
- S4.1.1: line 340 - are the two yaml environment files (stable and dev) distributed within the conda package? If so, this is bad practice, since it may lead to confusion on the part of the user; the conda package should contain files that are only from the stable released version, and if the user wants to use the development distribution, they should have the files associated with it when downloading (cloning) the gitball;
- S4.1.3: line 375: `e3sm_diags` executable - mention the word executable;
- S4.2: Zippy - is that a dependency of E3SM_Diags? Is it included with it as a sub-module? If it's a dependency, please include a link to its GH repo, or wherever the codebase lives, and to its documentation;
- same as above for S4.3: IICE
- Appendix A: line 460 - please don't use the copy-past call into a Python script - just say "execute/run the following Python script" (and it would be good if you could use e.g. Markdown colouring to simulate Python code colors); ideally try and make that an iPython Notebook - much more modern these days :)-
AC1: 'Reply on RC1', Chengzhu Zhang, 20 Aug 2022
The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2022-38/gmd-2022-38-AC1-supplement.pdf
-
AC1: 'Reply on RC1', Chengzhu Zhang, 20 Aug 2022
-
RC2: 'Comment on gmd-2022-38', Anonymous Referee #2, 22 Jul 2022
General comments
The paper provides a description of the E3SM Diags package. The package produces sets of plots comparing a E3S model run to observation or to another model run. It is used routinely in the E3SM model development cycle. The tool can also retrieve CMIP data files from the ESGGF which is a nice capability. The tool is flexible and it allows customization of the plots. The package is an open-source software written in python and available on github.
The authors provide various examples of the diagnostics produced by the E3SM Diags. All the diagnostics in the figures are directly coming from E3SM Diags for demonstration purposes.
The authors also provide a link to a complete example of E3SM Diags run for model versus obs. It would be nice to also provide a link to model versus model runs.
While the section 1-3 reads quite well, section 4 is a bit tedious. It provides a flurry of details about the installation, configuration and installation of the package. I personally think Section 4 could be summarized into a simpler overview and the “nuts of bolts” could be moved to an appendix.
Specific suggestions/corrections
- L85: two dots after workflow
- L90: define API (Application Programming Interface)
- L183: sentence incomplete ?
- L340: While it is clear that the E3SM diags could be easily run on supported machines, the portability of the tool on other machines (non DOE machines) is not discussed.
- L454: Link should be fixed: https://https://portal.nersc.gov/project/e3sm/e3sm_diags_v2.6_GMD/v2_6_0_all_sets_paper/viewer/.
-
AC2: 'Reply on RC2', Chengzhu Zhang, 20 Aug 2022
The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2022-38/gmd-2022-38-AC2-supplement.pdf
Status: closed
-
RC1: 'RC Comment on gmd-2022-38', Valeriu Predoi, 10 May 2022
Review also available at https://github.com/valeriupredoi/review_gmd-2022-38/blob/master/review_gmd-2022-38.md
Paper title: The E3SM Diagnostics Package (E3SM Diags v2.6): A Python-basedDiagnostics Package for Earth System Models Evaluation
Preprint: https://gmd.copernicus.org/preprints/gmd-2022-38/gmd-2022-38.pdfReview data
- Review by: Valeriu Predoi
- Date: 10 May 2022General remarks
The paper reads well, and contains about the right balance between technical vs scientific aspects, however, I have a couple suggestions:
- is this a technical overview work or a more scientifically-oriented description of the diags/metrics in E3SM_Diags? It'd be worth stating this clearly at the very top;
- the technical aspect opens the narrative, with a general description of the package's workflow, only to be interrupted by a very detailed scientific description of a number of standard metrics, and, after these metrics have been explored, the narrative comes back to technical aspects related to installation, I/O etc. It would be good to have the technical aspects bunched together, to allow the reader to follow through, and understand how the thing works; same for the metrics/diagnostics - probably better understood after the reader has insight into the compute workflow aspects of E3SM_Diags;
- versions matter: what is the purpose of describing E3SM_Diags v2 (actually, a very specific v2.6) - are there any major enhancements/refactorings/performance improvements from v1? Is there a paper describing v1, and if so, please link it? If it was me, I'd really drop the very specific v2.6 and instead refer to the tool with a more general v2 only;
- probably avoid including very specific script filenames (e.g. `postprocessing_E3SM_data_for_TC_analysis.sh`), these are documented in the code documentation already; the reader should get a bird's eye of the workflow without getting them into the nitty-gritty details;
- some information about the code license is needed (GPL, Apache etc)General suggestions
- title: E3SM Diags v2.6 is mentioned, but 2.6.1 on conda-forge https://anaconda.org/conda-forge/e3sm_diags
suggestion: either change to "2.6+" or don't specify the version in the title, or specify why 2.6 is a major
release even though the number doesn't suggest that way;
- Abstract: please provide a reference/link for AMWG diagnostics package;
- Introduction: line 27: "Most tools listed here are focused on the atmosphere" - not entirely true, ESMValTool
has a solid component for ocean model data evaluation; suggest changing to "Most tools listed here
have started with a main focus on the atmosphere, but evolved to analyze other ESM realms too"
- Introduction: line 32 - please provide a reference (methods paper, documentation and/or software repository) for AMWG
- Introduction: line 57 - "...project takes a distinct but related approach" - contradiction in terms
- Introduction: line 63 "Distribution of these Python packages is now mostly accomplished through Conda" - please provide
a reference to the Anaconda documentation/main page; also note that "conda" is the executable and the
package manager tool, but Anaconda is the organization and the actual package manager, so it's probably best to
use Anaconda in this context, maybe even "Anaconda/Miniconda"
- Introduction: line 72 - "This paper is a comprehensive description of E3SM Diags (as of version 2.6)" - anything special
about v2.6? Please briefly describe why the choice of version;
- Table 1: I am assuming the references will be available as active links in the final version of the paper?
(idem): ESMValTool: AR5 and AR6, please, also - ESMValTool is a PyPi package too https://pypi.org/project/ESMValTool/
- Optional: ESMValTool handles OBS datasets too, that are not necessarily CMIP-like file formats, via a process
of CMOR-standardization (CMORization), maybe with mentioning?
- Tech overview: line 76 - maybe provide a link to the installation section of your documentation?
- Tech overview: line 83: multi-threading, really, up to you if you want to change from multiprocessing to multi-threading
- Tech overview: general: it would be nice to provide links to various main sections of the package documentation
e.g. when talking about installation, configuration, running instructions
- S3.1: line 128: how is the reference grid chosen?
- S3.2: line 157: a very detailed explanation on how the (calibration) run works, but a rather frugal explanation of how CMIP data is retrieved from an ESGF node: "The built-in derived variable module takes in CMIP variables and handles variable name and unit conversions" - it'd be good to have this explained a bit better, and in a user-friendly way e.g. data specifications include: var name, mip, experiment name, frequency etc - with these parameters gathered from user input, the code goes and grabs data from the ESGF via standard-path DRS structure conventions etc
- S4.1.1: line 340 - are the two yaml environment files (stable and dev) distributed within the conda package? If so, this is bad practice, since it may lead to confusion on the part of the user; the conda package should contain files that are only from the stable released version, and if the user wants to use the development distribution, they should have the files associated with it when downloading (cloning) the gitball;
- S4.1.3: line 375: `e3sm_diags` executable - mention the word executable;
- S4.2: Zippy - is that a dependency of E3SM_Diags? Is it included with it as a sub-module? If it's a dependency, please include a link to its GH repo, or wherever the codebase lives, and to its documentation;
- same as above for S4.3: IICE
- Appendix A: line 460 - please don't use the copy-past call into a Python script - just say "execute/run the following Python script" (and it would be good if you could use e.g. Markdown colouring to simulate Python code colors); ideally try and make that an iPython Notebook - much more modern these days :)-
AC1: 'Reply on RC1', Chengzhu Zhang, 20 Aug 2022
The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2022-38/gmd-2022-38-AC1-supplement.pdf
-
AC1: 'Reply on RC1', Chengzhu Zhang, 20 Aug 2022
-
RC2: 'Comment on gmd-2022-38', Anonymous Referee #2, 22 Jul 2022
General comments
The paper provides a description of the E3SM Diags package. The package produces sets of plots comparing a E3S model run to observation or to another model run. It is used routinely in the E3SM model development cycle. The tool can also retrieve CMIP data files from the ESGGF which is a nice capability. The tool is flexible and it allows customization of the plots. The package is an open-source software written in python and available on github.
The authors provide various examples of the diagnostics produced by the E3SM Diags. All the diagnostics in the figures are directly coming from E3SM Diags for demonstration purposes.
The authors also provide a link to a complete example of E3SM Diags run for model versus obs. It would be nice to also provide a link to model versus model runs.
While the section 1-3 reads quite well, section 4 is a bit tedious. It provides a flurry of details about the installation, configuration and installation of the package. I personally think Section 4 could be summarized into a simpler overview and the “nuts of bolts” could be moved to an appendix.
Specific suggestions/corrections
- L85: two dots after workflow
- L90: define API (Application Programming Interface)
- L183: sentence incomplete ?
- L340: While it is clear that the E3SM diags could be easily run on supported machines, the portability of the tool on other machines (non DOE machines) is not discussed.
- L454: Link should be fixed: https://https://portal.nersc.gov/project/e3sm/e3sm_diags_v2.6_GMD/v2_6_0_all_sets_paper/viewer/.
-
AC2: 'Reply on RC2', Chengzhu Zhang, 20 Aug 2022
The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2022-38/gmd-2022-38-AC2-supplement.pdf
Chengzhu Zhang et al.
Chengzhu Zhang et al.
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 618 | 215 | 16 | 849 | 11 | 8 |
- HTML: 618
- PDF: 215
- XML: 16
- Total: 849
- BibTeX: 11
- EndNote: 8
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1