Evaluating the impact of task aggregation in workflows with shared resource environments: use case for the MONARCH application

Marciani, Manuel G.; Castrillo, Miguel; Utrera, Gladys; Acosta, Mario C.; Kinoshita, Bruno P.; Doblas-Reyes, Francisco

doi:10.5194/gmd-18-9709-2025

Articles | Volume 18, issue 23

https://doi.org/10.5194/gmd-18-9709-2025

© Author(s) 2025. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/gmd-18-9709-2025

© Author(s) 2025. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 18, issue 23

Development and technical paper

|

05 Dec 2025

Development and technical paper |

| 05 Dec 2025

Evaluating the impact of task aggregation in workflows with shared resource environments: use case for the MONARCH application

Manuel G. Marciani, Miguel Castrillo, Gladys Utrera, Mario C. Acosta, Bruno P. Kinoshita, and Francisco Doblas-Reyes

Data sets

Wrapper Impact Workloads and BSC Slurm Simulator Output of Dynamic Traces from CEA Curie Manuel G. Marciani https://doi.org/10.5281/zenodo.10623439

Full Results from Simulations for Static and Dynamic Workloads Using BSC Slurm Simulator Manuel G. Marciani https://doi.org/10.5281/zenodo.10818813

Wrapper Impact Workloads and BSC Slurm Simulator Output of Static Traces based on Data from LUMI Supercomputer Manuel G. Marciani https://doi.org/10.5281/zenodo.10624403

Interactive computing environment

Static Workload Results Analysis Scripts Manuel G. Marciani https://doi.org/10.5281/zenodo.12801377

Scripts and Files to Add Workflow to Curie Manuel G. Marciani https://doi.org/10.5281/zenodo.12801281

Docker Image of the Computational Earth Sciences Slurm Simulator Manuel G. Marciani https://doi.org/10.5281/zenodo.12801138

BSC Computational Earth Sciences Slurm Simulator Tools Manuel G. Marciani https://doi.org/10.5281/zenodo.12800999

Lumi Workload Analysis Script Manuel G. Marciani https://doi.org/10.5281/zenodo.12801326

Short summary

Earth System Model simulations are typically run on large, highly congested flagship computers using workflows. These workflows can consist of thousands of tasks. If these tasks are queued individually, the wait time can add up, resulting in a long response time. In this paper, we explore a technique for aggregating tasks into a single submission. We found that this simple technique reduced the time spent in the queue by up to 7 %.