Articles | Volume 19, issue 17
https://doi.org/10.5194/gmd-19-8501-2026
© Author(s) 2026. This work is distributed under the Creative Commons Attribution 4.0 License.
Task aggregation as a strategy to optimize Earth System Model workflows in HPC: assessing real scenarios with EC-Earth
Download
- Final revised paper (published on 11 Sep 2026)
- Preprint (discussion started on 21 Apr 2026)
Interactive discussion
Status: closed
Comment types: AC – author | RC – referee | CC – community | EC – editor | CEC – chief editor
| : Report abuse
-
RC1: 'Comment on egusphere-2026-539', Anonymous Referee #1, 27 May 2026
- AC1: 'Reply on RC1', Pablo Goitia, 25 Jun 2026
-
RC2: 'Comment on egusphere-2026-539', Anonymous Referee #2, 06 Aug 2026
- AC2: 'Reply on RC2', Pablo Goitia, 25 Aug 2026
Peer review completion
AR – Author's response | RR – Referee report | ED – Editor decision | EF – Editorial file upload
AR by Pablo Goitia on behalf of the Authors (25 Aug 2026)
Author's response
Author's tracked changes
Manuscript
ED: Publish as is (02 Sep 2026) by Dan Lu
AR by Pablo Goitia on behalf of the Authors (04 Sep 2026)
General comments
Task aggregation as a strategy to optimize Earth System Model workflows in HPC: assessing real scenarios with EC-Earth by P. Goitia et al. describes a specific feature of the autosubmit workflow software to wrap multiple task into a single (SLURM) job. The authors compare performance measurements with the focus on time to solution by submitting tasks as single jobs versus aggregating tasks into a larger job. The authors point out the benefits of task aggregation and show a gain in time to solution of up top 20%. As pointed out by the authors this technique as such is not new but has not been used in the context of climate modelling with its considerably complex workflows. The general idea is well described and discussed. The use cases (here EC-Earth run on three different HPC system) are relevant for the climate modelling community. The paper is not general in the sense that it does not provide a guideline for someone who would like to take autosubmit and apply this to a different workflow like a different numerical model to be run on yet another HPC system. The manuscript is concise and well written. Nevertheless, I stumbled over some repetitions which could be avoided for the sake of providing new information.
I well accept the authors choice of keeping the focus on task scheduling as such. Having said that, in this light I am missing some more in-depths discussion of the interrelation between queue waiting times, shortcomings of SLURM parameter adjustments and its bypassing with the task aggregation approach. While I understand the underlying problem and the proposed solution I wonder if this can also be approached by simply making a single task running for longer time, e.g. using chunks of 10 years rather than 1 month. Thus I am left with a bit of a doubt that task aggregation is the only solution for any Earth system model to reduce the time to solution.
Specific comments
Abstract
A matter of taste, but to me the abstract does not read as a short summary highlighting the most important findings but already provides very detailed information (e.g. congested machines). Task aggregation and queue time are mentioned before they are defined.
Introduction
The introduction already provides details of the methods. I propose to shift these to your section 3 (Methods).
L 68ff I propose to move the introduction to autosubmit after the introduction of other work (as state of the art) and thus address the mentioned problems mentioned rather than jumping back and forth between your approach and other work.
L 95ff The political impact of simulations with EC-Earth3 is not really relevant in this context.
L 99 I have had a hard time understanding why your solution is transversal (see my remark for your sec. 3)
L 102/103 Isn’t this sentence better placed in the next and final paragraph of the introduction? You assume that the reader is already familiar with “fair share”. Thus I wonder if its mentioning here is too early.
L 104 tells me what I have just read and can be deleted.
L 111ff This is just a plain repetition of what was said in line 105 and 106.
Background
L 155 I do not get why the backfill algorithm plays a negative role. It is just designed to fill gaps if possible to guarantee an efficient use of the whole system.
L 187 is of an account: please rephrase.
L 215ff Is the detailed description of EC-Earth3 really necessary in this context here?
L 226, 227 Repetition from the introdction
L 236 we tried: Did you succeed?
Methods
Why does a chunk size has to be 12 months? Why can’t it be 120? This would allow one to run 5 single jobs without the need of wrapping? Is there some limitation set by EC-Earth3 or any of the internal components which stand in the way? Are there any other advantages using chunk sizes of 12 and thus a wrapper size > 1 to reach the maximum allowed queue time?
How and where in the manuscript do you make use of Slurm parameters fair share, Level Fair Share, and raw usage that you collected? Earlier on, you explained Level Fair Share. How is this related to Fair Share and Raw usage mentioned in the figures. I am afraid that the reader gets a bit lost here. A bit more guidance would be nice.
Results and discussion
I interpret the differences in SYPD for the wrapped vs unwrapped cases as system fluctuations. The percentage and speedup with a precision of up to two decimals (even though true for these single cases) suggest an exactness which is questionable. I assume that these numbers are rather volatile and depend on the overall load of the system at a given time.
Figures 4, 5 and 6: The figure title is a repetition of the figure caption.
Conclusions
You described task aggregation of rather homogenous tasks. If other tasks like postprocessing were added would you expect similar benefits? Would there still be a benefit if all users on a congested system opted for task aggregation? How much could be compensated by adjusting the SLURM parameters? Wouldn’t it make more sense to use reservations to runs those long experiments.