ICON ComIn &ndash; The ICON Community Interface (ComIn version 0.1.0, with ICON version 2024.01-01)

Hartung, Kerstin; Kern, Bastian; Dreier, Nils-Arne; Geisbüsch, Jörn; Haghighatnasab, Mahnoosh; Jöckel, Patrick; Kerkweg, Astrid; Loch, Wilton Jaciel; Prill, Florian; Rieger, Daniel

doi:https://doi.org/10.5194/gmd-2024-135

Preprints

https://doi.org/10.5194/gmd-2024-135

Preprints

Submitted as: development and technical paper

20 Sep 2024

Submitted as: development and technical paper |

| 20 Sep 2024

Status: a revised version of this preprint was accepted for the journal GMD and is expected to appear here in due course.

ICON ComIn – The ICON Community Interface (ComIn version 0.1.0, with ICON version 2024.01-01)

Kerstin Hartung, Bastian Kern, Nils-Arne Dreier, Jörn Geisbüsch, Mahnoosh Haghighatnasab, Patrick Jöckel, Astrid Kerkweg, Wilton Jaciel Loch, Florian Prill, and Daniel Rieger

Abstract. In 2021, a team of developers from the Deutscher Wetterdienst (DWD), the German Aerospace Center (Deutsches Zentrum für Luft- und Raumfahrt, DLR), the German Climate Computing Center (Deutsches Klimarechenzentrum, DKRZ) and the Forschungszentrum Jülich (FZJ) started the Icosahedral Nonhydrostatic (ICON) Model Community Interface (ComIn) project: ICON ComIn is a library with multi-language support for connecting third-party modules ('plugins') to the ICON model using the dynamic loader of the operating system. ComIn is intended for a wide range of use cases, from the integration of simple diagnostic Python scripts to full chemistry models into ICON. ICON ComIn is distributed with the ICON model code under an open source license. Its application programming interface (API) provides a low barrier for code extensions to ICON and reduces the migration effort in response to new ICON releases. ComIn's main design principles are that it is lightweight, interoperable (Fortran, C/C++, Python) and flexible, and required changes in ICON are minimised. During the development of ComIn the ease of getting started and the experience during plugin development were guiding principles to provide a convenient tool. The extensive documentation and a variety of test and example plugins are results of this process.

This paper motivates the underlying design principles and provides some concrete reasoning for their selection. Further, current limitations are discussed and the vision for the future is presented.

Received: 11 Jul 2024 – Discussion started: 20 Sep 2024

Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this preprint. The responsibility to include appropriate place names lies with the authors.

Download & links

Preprint (PDF, 1102 KB)

Supplement (184 KB)

Download & links

Kerstin Hartung, Bastian Kern, Nils-Arne Dreier, Jörn Geisbüsch, Mahnoosh Haghighatnasab, Patrick Jöckel, Astrid Kerkweg, Wilton Jaciel Loch, Florian Prill, and Daniel Rieger

Status: closed

CEC1:
'Comment on gmd-2024-135: No compliance with the policy of the journal', Juan Antonio Añel, 29 Oct 2024

Dear authors,
Unfortunately, after checking your manuscript, it has come to our attention that it does not comply with our "Code and Data Policy".

https://www.geoscientific-model-development.net/policies/code_and_data_policy.html
You have archived the ComIn code on a git repository. However, your git repository and the servers where it is hosted are not suitable for scientific publication, as they do not comply with the necessary long-term archival requirements. Therefore, please publish your code in one of the appropriate repositories and reply to this comment with the relevant information (link and a permanent identifier for it (e.g. DOI)) as soon as possible, as we can not accept manuscripts in Discussions that do not comply with our policy. Therefore, the current situation with your manuscript is irregular.
Moreover, you have not published in a permanent repository the ICON version that you use here for tests. Please, do it too.
Please, note that if you do not fix this problem, we will have to reject your manuscript for publication in our journal.
Also, you must include the modified 'Code and Data Availability' section in a potentially reviewed manuscript, the DOI of the code.
Juan A. Añel

Geosci. Model Dev. Executive Editor

Citation: https://doi.org/10.5194/gmd-2024-135-CEC1
- AC1:
  'Reply on CEC1', Kerstin Hartung, 06 Nov 2024
  
  Dear Juan A. Añel,
  thank you for your message on the code availability section of our manuscript. I think this is a misunderstanding which arose due to poor phrasing. Indeed, the used ComIn version is published as part of the used ICON version with the DOI https://doi.org/doi:10.35089/WDCC/IconRelease01 at https://www.wdc-climate.de. This information is already given in the reference “ICON partnership (2024b)” but as this is not clear enough from the text, we will update the section accordingly, for example like
  “The version of ComIn used in this article is part of the ICON version 2024.01-01 (tag tags/icon-2024.01-01 in the ICON repository https://gitlab.dkrz.de/icon/icon and branch release-2024.01-public in the public ICON repository https://gitlab.dkrz.de/icon/icon-model), which is available with the DOI https://doi.org/doi:10.35089/WDCC/IconRelease01 at https://www.wdc-climate.de (ICON Partnership (2024b)).”
  I hope that I could convince you that our manuscript is in-line with the journal’s code and data policies. If that is the case we will prepare an updated version of the manuscript in response to the reviewer’s comments, including the above modification or something similar.
  Kind regards,
  
  Kerstin Hartung
  
  Citation: https://doi.org/10.5194/gmd-2024-135-AC1
  - CEC2: 'Reply on AC1', Juan Antonio Añel, 06 Nov 2024
    
    Dear authors,
    Unfortunately your reply does not address the issue. There was not misunderstandings in my previous comment. The https://www.wdc-climate.de servers are not a suitable repository to store the assets associated to your manuscript. It must be an acceptable repository from those that we list in our policy. Please, address the issue storing your code and data in one of them. They will give you a new DOI or permanent identifier. Then, please, reply to this comment with the information (link and permanent identifier). Also, remember to include this new information in potential new versions or your manuscript, and remove the current one.
    Again, note that we will have to reject your manuscript for publication if this irregular situation regarding the assets used to develop your manuscript continues.
    Juan A. Añel
    Geosci. Model Dev. Executive Editor
    
    Citation: https://doi.org/10.5194/gmd-2024-135-CEC2
    
    AC2: 'Reply on CEC2', Kerstin Hartung, 12 Nov 2024
    
    Dear Juan,
    I am afraid we still do not get your point. The code used for the manuscript is open source accessible under the given DOI (https://doi.org/doi:10.35089/WDCC/IconRelease01). The WDCC provided this DOI and therefore guarantees the long term storage (part of the DOI certification). The WDCC is also included in the Springer Nature list of possible DOI providers in the GMD code and data policy section.
    
    Why is this DOI not acceptable?
    Kind regards,
    
    Kerstin Hartung
    
    Citation: https://doi.org/10.5194/gmd-2024-135-AC2
    
    CC1: 'Reply on AC2', Kerstin Hartung, 12 Nov 2024
    
    Dear Juan A. Añel,
    I am very sorry I addressed you incorrectly in the previous message.
    Kind regards,
    
    Kerstin Hartung
    
    Citation: https://doi.org/10.5194/gmd-2024-135-CC1
RC1:
'Comment on gmd-2024-135', Anonymous Referee #1, 05 Nov 2024

The paper by Hartung et al. describes a new library that is developed to couple external software, ranging from Python diagnostics to ESM components, to the ICON model. The article provides an extensive description of the ComIn library’s capabilities, underlying motivations and an overview of potential applications that can benefit the ICON community. I think the paper is well structured, providing necessary details to describe the ComIn library and the reasoning behind the software design. I recommend this article for a publication in GMD, however some minor comments should be addressed.
Specific comments:
* L187-193: Could you specify in which cases plugins are executed asynchronously? Is this possible for couplers and I/O servers when used as plugins through ComIn v0.1.0 or do these tasks also rely on blocking callbacks such as for synchronous plugins?
* The authors cautiously point out that plugin implementation and the use of additional resources to run plugins are the user’s responsibility. Indeed, synchronous execution of a plugin can pause the ICON model for quite some time for a computationally inefficient plugin, which is indicated in the manuscript. On the contrary, the article does not really show evidence of efficient execution of ICON/ComIn for an “efficient plugin” (e.g. a simple monitoring Python script to generate a map or compute a mean field value). A test with ComIn disabled in ICON-NWP is described and shows that the ICON/ComIn implementation introduces very little overhead for ICON, which is an important result to stress. But the current test runs do not show the impact of activating ComIn for ICON. An additional testcase would be desirable to show how much of a typical ICON timestep is spent on running a fast plugin.
* L255: Compatibility between the ICON-ComIn versions is verified by ComIn but what about the ComIn-plugin(s) compatibility? Does the software development process guarantee API backward compatibility so that plugins can be used with newer versions of ICON/ComIn? If not, do users need to develop several versions of a plugin depending on the ICON version in use?
* Sect 3.4.2: It can be complex for a Python script developer to work with ICON data produced on an unstructured grid in a parallel setup. Providing data post-processing functionalities (or at least guidelines on how to proceed) would improve the user experience and facilitate ComIn adoption. Does ComIn provide functionalities (through its API or as plugin examples) to reconstruct fields globally and to regrid fields on the target grids of plugins? Could you clarify how plugin developers should work with ICON data provided by ComIn? Sect 5 provides a valuable list of possibilities offered by the library but lacks details on how users should proceed.
* L438-441: Prototyping with Python to develop parametrizations would first require a translation of the Fortran code into Python. This may not be straightforward, considering that the two programming languages handle efficient array computations very differently. It is unclear why switching languages would speed-up the model development process, not to mention the optimization process which can only be carried out in the model’s native language. Nevertheless, I wonder to what extend parametrization development could rather benefit from ComIn when carried out in Fortran and outside the model, given that the library provide access to input/output parametrization fields and separate program build.
* L459-462: Opening up the possibility of using or developing Python packages (AI, ML, …) to emulate parametrized processes should be of interest to the ICON community. I think this point should be put forward instead of L438-441.
* L434-437: The purpose of the evaluation and performance analysis referred to here is unclear. Do you employ the mentioned tools (a sentence to describe these tools is missing) to monitor the use of computing resources for the simulation job (computing time, memory usage, …)? Or do you intend to compute relevant metrics to analyze physical phenomena of the simulation (perhaps in the form of timeseries)? Could you also clarify the statement “a well set-up simulation does not need to be restarted after evaluation”?
* L160-167: You could add a statement emphasizing that the interface library approach depends mainly on the model data structure and the model execution flow.
* L185: Could you explain on how the dynamical loading works in ComIn? Does it require usage of specific dependency management tools and other libraries that were not previously part of ICON-NWP?
* L225: The goal of limiting software maintenance effort (ICON, ComIn) is a valuable one and this is mentioned several times. Could you provide an estimate of the number of lines added to ICON in response to the inclusion of ComIn v0.1.0? Or perhaps provide the fraction of lines of code that were modified/added for ComIn into ICON relative to the ICON codebase?
Minor comments:
* L16: host model model -> host model?
* L66: from each of these process -> processes?
* L478: you may want to add a definition or link for CMOR
* L507: adapted -> adopted?

Citation: https://doi.org/10.5194/gmd-2024-135-RC1
- AC3: 'Reply on RC1', Kerstin Hartung, 03 Dec 2024
  
  The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2024-135/gmd-2024-135-AC3-supplement.pdf
  
  Citation: https://doi.org/10.5194/gmd-2024-135-AC3
RC2:
'Comment on gmd-2024-135', Anonymous Referee #2, 05 Nov 2024

The article details a plugin functionality added to the ICON model, based on a common adaptor library to be used by both the ICON model and the plugins. Entry point hooks are inserted in the ICON model to execute the plugins at various places.
The authors introduce the concepts in a structured manner comparing all possible approaches to extend models with external components. The described code is open-source and readily available to explore, including documentation and examples. Various conceivable scenarios are considered in how plugins operate with the given ICON data fields.
The article fits well in the context of this journal and is of high quality. I therefore recommend it for publication.
One question. Given that it is possible for plugins to share the data pointers with ICON, is there an optional safety mechanism provided (e.g. by checksumming the data) in order to ensure or verify that no data has been modified inadvertently?

Citation: https://doi.org/10.5194/gmd-2024-135-RC2
- AC4: 'Reply on RC2', Kerstin Hartung, 03 Dec 2024
  
  The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2024-135/gmd-2024-135-AC4-supplement.pdf
  
  Citation: https://doi.org/10.5194/gmd-2024-135-AC4

Status: closed

CEC1:
'Comment on gmd-2024-135: No compliance with the policy of the journal', Juan Antonio Añel, 29 Oct 2024

Dear authors,
Unfortunately, after checking your manuscript, it has come to our attention that it does not comply with our "Code and Data Policy".

https://www.geoscientific-model-development.net/policies/code_and_data_policy.html
You have archived the ComIn code on a git repository. However, your git repository and the servers where it is hosted are not suitable for scientific publication, as they do not comply with the necessary long-term archival requirements. Therefore, please publish your code in one of the appropriate repositories and reply to this comment with the relevant information (link and a permanent identifier for it (e.g. DOI)) as soon as possible, as we can not accept manuscripts in Discussions that do not comply with our policy. Therefore, the current situation with your manuscript is irregular.
Moreover, you have not published in a permanent repository the ICON version that you use here for tests. Please, do it too.
Please, note that if you do not fix this problem, we will have to reject your manuscript for publication in our journal.
Also, you must include the modified 'Code and Data Availability' section in a potentially reviewed manuscript, the DOI of the code.
Juan A. Añel

Geosci. Model Dev. Executive Editor

Citation: https://doi.org/10.5194/gmd-2024-135-CEC1
- AC1:
  'Reply on CEC1', Kerstin Hartung, 06 Nov 2024
  
  Dear Juan A. Añel,
  thank you for your message on the code availability section of our manuscript. I think this is a misunderstanding which arose due to poor phrasing. Indeed, the used ComIn version is published as part of the used ICON version with the DOI https://doi.org/doi:10.35089/WDCC/IconRelease01 at https://www.wdc-climate.de. This information is already given in the reference “ICON partnership (2024b)” but as this is not clear enough from the text, we will update the section accordingly, for example like
  “The version of ComIn used in this article is part of the ICON version 2024.01-01 (tag tags/icon-2024.01-01 in the ICON repository https://gitlab.dkrz.de/icon/icon and branch release-2024.01-public in the public ICON repository https://gitlab.dkrz.de/icon/icon-model), which is available with the DOI https://doi.org/doi:10.35089/WDCC/IconRelease01 at https://www.wdc-climate.de (ICON Partnership (2024b)).”
  I hope that I could convince you that our manuscript is in-line with the journal’s code and data policies. If that is the case we will prepare an updated version of the manuscript in response to the reviewer’s comments, including the above modification or something similar.
  Kind regards,
  
  Kerstin Hartung
  
  Citation: https://doi.org/10.5194/gmd-2024-135-AC1
  - CEC2: 'Reply on AC1', Juan Antonio Añel, 06 Nov 2024
    
    Dear authors,
    Unfortunately your reply does not address the issue. There was not misunderstandings in my previous comment. The https://www.wdc-climate.de servers are not a suitable repository to store the assets associated to your manuscript. It must be an acceptable repository from those that we list in our policy. Please, address the issue storing your code and data in one of them. They will give you a new DOI or permanent identifier. Then, please, reply to this comment with the information (link and permanent identifier). Also, remember to include this new information in potential new versions or your manuscript, and remove the current one.
    Again, note that we will have to reject your manuscript for publication if this irregular situation regarding the assets used to develop your manuscript continues.
    Juan A. Añel
    Geosci. Model Dev. Executive Editor
    
    Citation: https://doi.org/10.5194/gmd-2024-135-CEC2
    
    AC2: 'Reply on CEC2', Kerstin Hartung, 12 Nov 2024
    
    Dear Juan,
    I am afraid we still do not get your point. The code used for the manuscript is open source accessible under the given DOI (https://doi.org/doi:10.35089/WDCC/IconRelease01). The WDCC provided this DOI and therefore guarantees the long term storage (part of the DOI certification). The WDCC is also included in the Springer Nature list of possible DOI providers in the GMD code and data policy section.
    
    Why is this DOI not acceptable?
    Kind regards,
    
    Kerstin Hartung
    
    Citation: https://doi.org/10.5194/gmd-2024-135-AC2
    
    CC1: 'Reply on AC2', Kerstin Hartung, 12 Nov 2024
    
    Dear Juan A. Añel,
    I am very sorry I addressed you incorrectly in the previous message.
    Kind regards,
    
    Kerstin Hartung
    
    Citation: https://doi.org/10.5194/gmd-2024-135-CC1
RC1:
'Comment on gmd-2024-135', Anonymous Referee #1, 05 Nov 2024

The paper by Hartung et al. describes a new library that is developed to couple external software, ranging from Python diagnostics to ESM components, to the ICON model. The article provides an extensive description of the ComIn library’s capabilities, underlying motivations and an overview of potential applications that can benefit the ICON community. I think the paper is well structured, providing necessary details to describe the ComIn library and the reasoning behind the software design. I recommend this article for a publication in GMD, however some minor comments should be addressed.
Specific comments:
* L187-193: Could you specify in which cases plugins are executed asynchronously? Is this possible for couplers and I/O servers when used as plugins through ComIn v0.1.0 or do these tasks also rely on blocking callbacks such as for synchronous plugins?
* The authors cautiously point out that plugin implementation and the use of additional resources to run plugins are the user’s responsibility. Indeed, synchronous execution of a plugin can pause the ICON model for quite some time for a computationally inefficient plugin, which is indicated in the manuscript. On the contrary, the article does not really show evidence of efficient execution of ICON/ComIn for an “efficient plugin” (e.g. a simple monitoring Python script to generate a map or compute a mean field value). A test with ComIn disabled in ICON-NWP is described and shows that the ICON/ComIn implementation introduces very little overhead for ICON, which is an important result to stress. But the current test runs do not show the impact of activating ComIn for ICON. An additional testcase would be desirable to show how much of a typical ICON timestep is spent on running a fast plugin.
* L255: Compatibility between the ICON-ComIn versions is verified by ComIn but what about the ComIn-plugin(s) compatibility? Does the software development process guarantee API backward compatibility so that plugins can be used with newer versions of ICON/ComIn? If not, do users need to develop several versions of a plugin depending on the ICON version in use?
* Sect 3.4.2: It can be complex for a Python script developer to work with ICON data produced on an unstructured grid in a parallel setup. Providing data post-processing functionalities (or at least guidelines on how to proceed) would improve the user experience and facilitate ComIn adoption. Does ComIn provide functionalities (through its API or as plugin examples) to reconstruct fields globally and to regrid fields on the target grids of plugins? Could you clarify how plugin developers should work with ICON data provided by ComIn? Sect 5 provides a valuable list of possibilities offered by the library but lacks details on how users should proceed.
* L438-441: Prototyping with Python to develop parametrizations would first require a translation of the Fortran code into Python. This may not be straightforward, considering that the two programming languages handle efficient array computations very differently. It is unclear why switching languages would speed-up the model development process, not to mention the optimization process which can only be carried out in the model’s native language. Nevertheless, I wonder to what extend parametrization development could rather benefit from ComIn when carried out in Fortran and outside the model, given that the library provide access to input/output parametrization fields and separate program build.
* L459-462: Opening up the possibility of using or developing Python packages (AI, ML, …) to emulate parametrized processes should be of interest to the ICON community. I think this point should be put forward instead of L438-441.
* L434-437: The purpose of the evaluation and performance analysis referred to here is unclear. Do you employ the mentioned tools (a sentence to describe these tools is missing) to monitor the use of computing resources for the simulation job (computing time, memory usage, …)? Or do you intend to compute relevant metrics to analyze physical phenomena of the simulation (perhaps in the form of timeseries)? Could you also clarify the statement “a well set-up simulation does not need to be restarted after evaluation”?
* L160-167: You could add a statement emphasizing that the interface library approach depends mainly on the model data structure and the model execution flow.
* L185: Could you explain on how the dynamical loading works in ComIn? Does it require usage of specific dependency management tools and other libraries that were not previously part of ICON-NWP?
* L225: The goal of limiting software maintenance effort (ICON, ComIn) is a valuable one and this is mentioned several times. Could you provide an estimate of the number of lines added to ICON in response to the inclusion of ComIn v0.1.0? Or perhaps provide the fraction of lines of code that were modified/added for ComIn into ICON relative to the ICON codebase?
Minor comments:
* L16: host model model -> host model?
* L66: from each of these process -> processes?
* L478: you may want to add a definition or link for CMOR
* L507: adapted -> adopted?

Citation: https://doi.org/10.5194/gmd-2024-135-RC1
- AC3: 'Reply on RC1', Kerstin Hartung, 03 Dec 2024
  
  The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2024-135/gmd-2024-135-AC3-supplement.pdf
  
  Citation: https://doi.org/10.5194/gmd-2024-135-AC3
RC2:
'Comment on gmd-2024-135', Anonymous Referee #2, 05 Nov 2024

The article details a plugin functionality added to the ICON model, based on a common adaptor library to be used by both the ICON model and the plugins. Entry point hooks are inserted in the ICON model to execute the plugins at various places.
The authors introduce the concepts in a structured manner comparing all possible approaches to extend models with external components. The described code is open-source and readily available to explore, including documentation and examples. Various conceivable scenarios are considered in how plugins operate with the given ICON data fields.
The article fits well in the context of this journal and is of high quality. I therefore recommend it for publication.
One question. Given that it is possible for plugins to share the data pointers with ICON, is there an optional safety mechanism provided (e.g. by checksumming the data) in order to ensure or verify that no data has been modified inadvertently?

Citation: https://doi.org/10.5194/gmd-2024-135-RC2
- AC4: 'Reply on RC2', Kerstin Hartung, 03 Dec 2024
  
  The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2024-135/gmd-2024-135-AC4-supplement.pdf
  
  Citation: https://doi.org/10.5194/gmd-2024-135-AC4

Kerstin Hartung, Bastian Kern, Nils-Arne Dreier, Jörn Geisbüsch, Mahnoosh Haghighatnasab, Patrick Jöckel, Astrid Kerkweg, Wilton Jaciel Loch, Florian Prill, and Daniel Rieger

Supplement

https://doi.org/10.5194/gmd-2024-135-supplement

Kerstin Hartung, Bastian Kern, Nils-Arne Dreier, Jörn Geisbüsch, Mahnoosh Haghighatnasab, Patrick Jöckel, Astrid Kerkweg, Wilton Jaciel Loch, Florian Prill, and Daniel Rieger

Viewed

Total article views: 633 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	Supplement	BibTeX	EndNote
379	140	114	633	26	9	11

HTML: 379
PDF: 140
XML: 114
Total: 633
Supplement: 26
BibTeX: 9
EndNote: 11

Views and downloads (calculated since 20 Sep 2024)

Month	HTML	PDF	XML	Total
Sep 2024	89	7	24	120
Oct 2024	65	17	75	157
Nov 2024	170	92	13	275
Dec 2024	55	24	2	81

Cumulative views and downloads (calculated since 20 Sep 2024)

Month	HTML	PDF	XML	Total
Sep 2024	89	7	24	120
Oct 2024	65	17	75	157
Nov 2024	170	92	13	275
Dec 2024	55	24	2	81

Viewed (geographical distribution)

Total article views: 615 (including HTML, PDF, and XML) Thereof 615 with geography defined and 0 with unknown origin.

Country	#	Views	%

Latest update: 13 Dec 2024

Short summary

The Icosahedral Nonhydrostatic (ICON) Model Community Interface (ComIn) library supports connecting third-party modules to the ICON model. Third-party modules can range from simple diagnostic Python scripts to full chemistry models. ComIn offers a low barrier for code extensions to ICON, provides multi-language support (Fortran, C/C++ and Python) and reduces the migration effort in response to new ICON releases. This paper presents the ComIn design principles and a range of use cases.


Total:	0
HTML:	0
PDF:	0
XML:	0