Improving ocean modeling software NEMO 4.0 benchmarking and communication efficiency

Irrmann, Gaston; Masson, Sébastien; Maisonnave, Éric; Guibert, David; Raffin, Erwan

doi:https://doi.org/10.5194/gmd-15-1567-2022

Articles | Volume 15, issue 4

https://doi.org/10.5194/gmd-15-1567-2022

© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.

Special issue:

Nucleus for European Modelling of the Ocean - NEMO

https://doi.org/10.5194/gmd-15-1567-2022

© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 15, issue 4

Development and technical paper

|

22 Feb 2022

Development and technical paper |

| 22 Feb 2022

Improving ocean modeling software NEMO 4.0 benchmarking and communication efficiency

Gaston Irrmann, Sébastien Masson, Éric Maisonnave, David Guibert, and Erwan Raffin

Download

Final revised paper (published on 22 Feb 2022)
Preprint (discussion started on 21 Oct 2021)

Interactive discussion

Status: closed

RC1:
'Comment on gmd-2021-320', Anonymous Referee #1, 01 Nov 2021

Review of MS No. gmd-2021-320 “Improve Ocean Modelling Software NEMO 4.0 benchmarking and communication efficiency” by Gaston Irrmann et al.

Improving the computational efficiency of open-source community ocean models, such as NEMO v4.0, is of great benefits to climate modelling and operational ocean forecasting. This manuscript reports recent contributions through the collaboration of numerical modelling and HPC experts in several aspects: (1) the creation of a benchmark (BENCH) configuration for testing the model efficiency; (2) the automatic definition of “optimal” domain decomposition; and (3) code modifications to reduce the communications among subdomains in two bottleneck routines. I believe the work described in this manuscript are valuable for both model developers and users.

In my review, I did not dig in all the technical details described in this manuscript because I am not an expert in parallel computing. For this, please consider this as a partial review. My focus is on the presentation aspect, as I find the manuscript is rather difficult to read due mainly to the writing style that makes the presentation neither concise nor accurate. There are many long sentences that are not easy to follow. There are also numerous grammar mistakes or typos throughout the manuscript. I will give some examples in the following “Specific comments” and “Technical corrections”, but the list is far from being complete. I suggest that the manuscript needs a thorough revision/editing before being accepted for publication in a journal for broader readers.

L39-41: Due to the way of writing, many sentences in the manuscript have unclear meanings that are difficult for a reader to understand. Here is an example: “This is why the incremental approach we follow is addressing and facilitating the future addressing of the problem of inter and intra node MPI communication cost. We assume that its reduction is and will stay worthwhile, independently of hardware evolution”. Here, I find it is hard to follow the exact meaning of “incremental approach”, “future addressing”, “its reduction” and “is and will stay”. Here is my attempt to revise these sentences: “Thus, the work of this study focuses on the reduction the MPI communication cost, for both inter-mode and intra-node. We expect that the need for such reduction will continue in future, independent of hardware evolution”. Do authors agree that this makes easier to read? If yes, I suggest they make the effort to check and revise the whole manuscript.

:

L5: “straights” should be “straight”

L14: “tests” should be “test”

L15: quantify

L28: “constant renewal of the model equations” may be rephrased as “constant improvement of the model physics”

L60: “In their figure 4” or “In their Figure 4”? Please check the journal format, and use the same format throughout the manuscript.

L82: “lower or equal than” to “lower than or equal to”

L85: “Note that we could imagine that in a few cases”, can you simply write “Note that in a few cases”?

L103: “Until revision 3.6 included” meaning is not clear

L108-116: “process” and “processes” - can the wording be revised to make the meaning more clear?

L162: “in (Maisonnave and Masson, 2019)” should be “in Maisonnave and Masson (2019)”

L172: extra “the”

L175: level(s), let(s)

L184: “can be used band be used”?

L188: “in such a was”?

L189: “corresponds approximately of (to)”

L195: “have” be “has”

L214: “prevent to test is”?

L269-270: “we” or “they”?

L339: “with use here with with”?

L342: “on the second an before”?

L400: “are diminished by a third”?

Fig.8 caption: “North-Atlantic”, somewhere in text is ‘West Atlantic”

Title of section 4: “Conclusions and discussion”

L481: “is maybe”?

L494: “strait” be “straight”?

Citation: https://doi.org/10.5194/gmd-2021-320-RC1
- AC1: 'Reply on RC1', Gaston Irrmann, 15 Dec 2021
  
  Thank you for your input and corrections. We have taken them into account and revised the whole manuscript, changing formulations to improve conciseness and clearing ambiguities. We hope you will find this new version of our manuscript clearer and easier to read.
  
  Citation: https://doi.org/10.5194/gmd-2021-320-AC1
- AC2: 'Reply on RC1', Gaston Irrmann, 15 Dec 2021
  
  Thank you for your input and corrections. We have taken them into account and revised the whole manuscript, changing formulations to improve conciseness and clearing ambiguities. We hope you will find this new version of our manuscript clearer and easier to read.
  
  Citation: https://doi.org/10.5194/gmd-2021-320-AC2
RC2:
'Comment on gmd-2021-320', Anonymous Referee #2, 22 Nov 2021
The mansucript describes work that is of interst to the broad NEMO user community. Authors develop a new benchmarking framework, implement an automatic optimized domain decomposition and reduce the communication footprint in two bottleneck routines.

Technical/Scientific comments:

The authors point out that the scalability of global configuration model is limited by the load imbalance due to the special halo exchange required for the north fold. This fact does not seem to be reflected in the design of the domain decomposition algorithm. Would it be appropriate to attempt to address the load imbalance at this stage. Does the proposed domain decomposition algorithm lead to improved load balance? It would be desirable to supplement the text with figure illustrating an example of the algorithm in action.

The effort to group communications togeather in one case or remove altogeather in the other translates to significant compuational gains. Figure 8 shows a frequent occurance of abnormally slow time steps, with the factor of up to x10. Is it possible to verify that the slowdown occurs for random cores or perhaps is it systematic for a paritcular core. I would suggest to run the tests of a multiplicity of full electrical groups to avoid other jobs affect the network congestion in your experiments.

Other comments:

I suggest changing the title to: "Improving Ocean Modelling Software ..";

L3: ".. actual 4.0 reference .." - what does 'actual' refer to here? wouldn't it be clear to just write ".. 4.0 reference ..";

L28: ".. developed in a sustainable way by a European consortium .." - what does "sustainable way" mean here, how is it defined? Sustainable in what sense?

L28: "The constant renewal of the model equations .." - I would suggest to use the term revision or reformulation here;

L33: ".. and are not necessary experts .." -> ".. and are not necessarilly experts ..";

L36: ".. the actual reference version of the code" - remove "actual";

L51/2: ".. to provide a proper benchmarking environment" - remove "proper".
Citation: https://doi.org/10.5194/gmd-2021-320-RC2
- AC3: 'Reply on RC2', Gaston Irrmann, 15 Dec 2021
  
  Thank you for your review and comments. We have changed the manuscript as to address your observations.
  As you noticed, the domain decomposition, described in this paper, does not directly take into account the load increase due to the special halo exchange for the north pole. We have nevertheless introduced an optimization for this north pole issue that was not described in the original version of our manuscript. We added to the paper a detailed explanation of what we do that has been summarised in the following text :
  When we look for the optimal domain decomposition, the choice of the number of lines and columns used to split the domain into the MPI subdomains we test, is done without considering the possible existence of the north pole folding in the configuration chosen by the user.
  
  As we saw, the Euclidean division used to define the MPI subdomain may have a non-zero remainder (r), which forces some MPI subdomains to have one more point in the i/j directions than the other ones. This is where we added our optimization for configurations requiring the north pole folding.
  
  Instead of cutting the domain with the following the following default j-decomposition: r ∗ jpj_max + (jpnj-r)∗(jpj_max-1), we use the following j-decomposition: (jpnj-1)∗jpj_max + jpj_north with jpj_north = jpj_max - jpnj + r.
  
  We hence reduce the size of the northernmost MPI subdomains and improve the load balance.
  
  To investigate the issue regarding abnormally slow time steps we had to break down the timing much further.
  
  Indeed, the slowdown originates on a single core of the simulation at a very fine time scale. Because of the communications, by the end of the time step the delay produced by a slower core will be propagated to a significant portion of the simulation. We hence timed each call to an MPI routine inside NEMO's communication routine and the timed the execution in between MPI calls.
  On the figure A attached, on each plot the horizontal axis is the index of the call to the communication routine in the time step, on the vertical axis is the execution time on a logarithmic scale. The different figures correspond to : (top, left to right) the East-West communication, small computations following that communication, the North-South communication, (bottom, left to right) small computations and deallocation following the North-South communication, the computations following the whole communication (the code outside the communication routine) and the total time spent in the communication routine. For each figure the percentiles computed over the course of a simulation of 2080 time steps are plotted in colors and the average in bold black. In this figure, the measurement is done on the $39^{th}$ allocated core (MPI rank), the positions of the core is highlighted in the top middle graph. This core was chosen at random but all the cores of the simulation yield very similar results.
  East-West communications are intra-node and do not involve interconnect network. As even for intra-node communications the maximum of the execution time is very high, the location of the nodes even on distant parts of the machine is not the cause of the slowdowns. The cores responsible for the slowdowns are also seemingly random as each core seems to behave in a similar way. As the slowdown occurs on random cores it is likely to be caused by the hardware or OS interference. We have tried on several supercomputers and still found this issue, occurring more often on some than others. We are currently investigating ways to make NEMO more resilient to such slowdowns.
  The article was revised very concisely by adding the following sentence to the part about outliers : "A finer analysis showed that the slowdowns occur on random cores of the simulation."
  
  Citation: https://doi.org/10.5194/gmd-2021-320-AC3

Peer review completion

AR: Author's response | RR: Referee report | ED: Editor decision | EF: Editorial file upload

AR by Gaston Irrmann on behalf of the Authors (06 Jan 2022) Author's response Author's tracked changes Manuscript

ED: Publish subject to technical corrections (08 Jan 2022) by Xiaomeng Huang

AR by Gaston Irrmann on behalf of the Authors (14 Jan 2022) Manuscript

Short summary

To be efficient on supercomputers, software must be high-performance at computing many concurrent tasks. Communications between tasks is often necessary but time consuming, and ocean modelling software NEMO 4.0 is no exception. In this work we describe approaches enabling fewer communications, an optimization to share the workload more equally between tasks and a new flexible configuration to assess NEMO's performance easily.