High-impact Earth System Model (ESM) simulations are normally executed in cutting-edge High Performance Computing (HPC) resources. In these environments, they typically entail several steps, including model compilation, data movement, model execution, and post-processing. Furthermore, the execution of the model is often divided into multiple segments, which we refer to as “chunks” of simulated time, which is done to provide natural checkpoints for the simulations and to comply with maximum execution time constraints imposed by the system administrators (i.e., the wallclock). This results in workflows of up to thousands of individual tasks – the workflow's indivisible unit of compute –, as is the case with Destination Earth digital twin's . To handle all of this complexity, ESM simulations are automated with workflows.

Given the high cost associated with running such simulations, the Earth Sciences community has always been looking for ways of reducing the time-to-solution, which is the total time spanning from the first submission to the finish of the latest. The traditional front is to minimize the execution time of the most computationally intensive code within the model, such as the work of .

However, lately the community has drawn attention to the efficiency of the simulations, taking into account not only the runtime of the simulation but also the time spent with the postprocessing, failures, and in the queue. With this in mind, proposed a set of performance metrics for Earth system model simulations. Among these metrics, the authors proposed the simulated years per day (SYPD), which is the ratio of the time simulated in years with respect to the runtime of the simulation in days, as well as the actual simulated years per day (ASYPD), which is the simulated time in years divided by the time-to-solution of the simulation, therefore also accounting for time in queue and failures.

In , the authors computed these metrics for 33 CMIP6 simulations executed on 14 machines. Their analysis showed that the difference between ASYPD and SYPD ranged from 0 % to 78 %. But, they noted that not all institutions reported ASYPD consistently. Some accounted for both interruptions and queue time, while others accounted only for queue time. For those institutions that only accounted for queue time, the spread was between 10 % and 20 %. The authors therefore concluded “that queuing time represents an increment of around 10 %–20 % of the speed of the ESM”.

The software that handles the queue of resource requests in HPC platforms is called workload manager, or scheduler. We call each resource request a job. The development of schedulers is an active area of research within the HPC scheduler community, focusing on optimizing job scheduling policies to minimize queue time, while keeping a high utilization of the machine . Some examples of these policies are: first-come, first-served, least processing time, and priority-based. This last one is called multifactor in the ubiquitous Slurm workload manager .

In Slurm, under the multifactor policy, jobs are ordered decreasingly by an integer called “priority”, which is computed from “factors” that come from the job request (“size”, “partition”, “QoS”), time in queue (“age”), and the user's “fair share”. The purpose of this last factor, which is typically prioritized by system administrators, is to balance the responsiveness of the machine by favoring users who have not used their quota of the resources while penalizing those who overuse the machine. Besides the job's priority, Slurm also aims to maximize system utilization by implementing a technique called backfill . This technique allows jobs with less priority to use free spots in the machine, as long as they do not interfere with the jobs ahead of them.

Observing the impact of fair share on the priority, the developers of Autosubmit , a Workflow Manager System (WfMS) designed for climate and air-quality applications, have implemented a task aggregation technique called “wrappers”. The idea is that Autosubmit aggregates, or wraps, multiple subsequent tasks into a single job submission to save the users from queuing multiple jobs under a low priority. No changes are done to the tasks, therefore the natural checkpointing is still maintained, and the dependencies are ensured among the tasks. Additionally, wrapping is also employed for concurrent tasks in order to comply with the maximum number of jobs in the queue.

Aggregation was implemented in other fields, with different degrees of sophistication. In Earth Sciences, suggested using Cylc's feature to submit multiple jobs to reduce the queue time. Both Aiida and Snakemake – from the material and life sciences fields, respectively – provide a way of submitting multiple workflow tasks as a single job. The former implements this via a “metascheduler” plugin (HyperQueue plugin), while the latter refers to aggregation as “grouping”.

Moreover, pilot-job systems were developed to increase workflow throughput with more sophisticated solutions . These systems are characterized by implementing “resource placeholders, multi-level scheduling, and coordination patterns to enable task-level distribution and parallelism on multi-tenant resources”. One major example of a modern pilot-job system is Radical-PILOT .

Wrappers share some of the features of a typical pilot-job system. Wrappers provide a simpler “resource placeholder”, where all the task requests are added into a large submission. However, their objective is to either increase throughput by reducing submissions or to comply with maximum job restrictions. Therefore, besides fault tolerance, wrappers do not improve task scheduling and coordination within the allocation.

Currently, in our field, wrappers are used to reduce the queue time in the MONARCH application, in the climate model EC-Earth3 . They are also employed in the Destination Earth digital twin, to both reduce queue time and to comply with the platforms' policy regarding maximum jobs queuing per user.

Although aggregation is utilized by Autosubmit users and also in other fields, with positive impact reported, there is a lack of understanding of the reasons and conditions under which aggregating reduces queue time. Therefore, in this work, we tested whether wrapping subsequent tasks together reduces queue time and if the fair share is the most important factor in reducing queue time.

To test both theses, we measured queue time by simulating the HPC environment using a Slurm simulator . We modeled a congested workload of an HPC platform and use the workflow of the MONARCH air-quality application. Then, we ran under the same conditions a simulation with wrappers and without.

We found that combining tasks into one submission reduces the queue time by at most 7 % of the total workflow runtime, and we observe gains across all the tested fair share values. Moreover, this reduction can be greater for longer workflows due to the many times less tasks being submitted. In the context of the CMIP6 runs of the IS-ENES3, this reduction equates to at least eight fewer days of queue time for half of the runs. Additionally, we found a high inverse correlation of -0.87 between the fair share factor and time in the queue.

Our study provides a quantification of the reduction of queue time achieved by aggregating tasks of air quality applications that use HPC resources on congested platforms. This approach can also be applied to other simulations with temporal constraints that require a large amount of resources for a sustained time. Our results help to advance the understanding, from the user side, of how to optimize the submission in order to reduce the total queue time of their workflows.

This paper is organized as follows: the background (Sect. ) defines wrapping and introduces major points of the scheduling policy of Slurm. The methodology (Sect. ) provides a detailed explanation of the experimentation. This includes how we introduce the workflow onto the workload files, how we control the fair share, and the software we used and implemented to run the simulations. The results section (Sect. ) exposes the output of the experiments. This leads to the discussion (Sect. ) chapter, where we comment our findings and the relation between fair share and queue time. Finally, in the conclusions, we evaluate if our findings support our theses.

In this section, we start with an overview on Slurm's scheduling in Sect.  because of its wide adoption across HPC platforms. Then, we explain why and how wrappers are used in Sect. .

In this section, we explain the methodology employed in this study. First, in Sect. , we provide a description of the HPC platforms we simulate and the hardware we used to run the simulations. Then we give an overview of the simulator in Sect. . We cover the scheduling policy utilized in our experimentation in Sect. . In Sect. , we explain how we carried out simulations with synthetic static workloads modeled after the supercomputer to address the modern job demands. In Sect. , we ran a dynamic simulation utilizing a real workload from a long decommissioned system to have a representative daily usage pattern. Finally, for modeling the workflow, we use the allocated CPUs and runtime of the job running the Nonhydrostatic Multiscale Model on the B-grid (NMMB) within the MONARCH application .

We perform a two-fold experimentation because we wanted to capture both a representative daily usage pattern (arrival times) and modern HPC job demands (requested CPUs, wallclock, user, and group identification).

In this section, we analyze the results of the runs of both types of experiments: the static workloads in Sect.  and the dynamic workload in Sect. .

Full results tables were omitted due to space constraints, but they are available in the Zenodo platform .

As seen in Fig. , we achieved a reduction in queue time across all fair share values in the dynamic results. On average, this reduction was 1 %, reaching up to a 7 % decrease in queue time relative to the total workflow runtime. These results support the hypothesis that the reduction is caused by avoiding multiple submissions.

Since we observed consistent (on average and on median) reductions when using aggregation, we anticipate greater gains in longer workflows running in congested environments because bigger wrappers can be created, reducing the number of submissions and therefore the time spent waiting for resources.

Additionally, the 7 % figure could be greater if we consider that the machine had only two days of congestion per week. Current flagship systems are usually congested, and it is not uncommon for jobs to queue for days.

We observed three negative outliers, that is, the instances where the unwrapped workflow stood less in queue than the wrapped. We believe those were particular instances where the combination of running jobs and queuing jobs allowed the backfill algorithm to schedule the separate tasks earlier because they were smaller. Under this scenario, as we increase the length of the job, it is less likely that the scheduler would have free spots to execute, so the job has to wait to be scheduled by its priority. But, given the few instances where we observed arrangement, we believe this scenario is unlikely.

As for the static results, where we captured modern job requests (i.e., the requested CPUs and wallclock) modeled after the LUMI supercomputer, these give us an idea of what are the meaningful factors that immediately impact after submission under a stressed environment. What we see is a total correlation of the queue time with respect to the fair share, and, even stronger, an exponential relationship. This means that with a fair share of just 0.5, it is enough to have more priority than all the other jobs in the queue, and therefore to be scheduled immediately.

This exponential relation comes from the way we set the fair share to all the synthetically generated users. Since their prior usage was set to be proportional to the generated workload, their fair share followed the log-normal distribution.

Regarding job requests in static executions, we observe in Table  that both runtime and allocated CPUs are independent of queue time. This is explained because these simulations are short compared with the dynamic ones, not allowing for the age factor to build up, and neither the backfill algorithm to be noticeable.

Finally, Fig.  shows the expected linear relationship between priority and fair share. This is due to the weighted sum that Slurm uses to compute the priority in Eq. (). The slope of the fitted line, 84 374, nearly matches the fair share weight of the Slurm configuration that we used, which is 100 000.

As is the case for the queue time, the requested CPUs and runtime are not correlated with the priority. This is due to the aforementioned shortness of the simulations and the relatively small application request with respect to the large system. Taking the maximum time that any of our tracked jobs was in the queue, 372 s, adds just 61 (372 divided by 7 d in seconds multiplied by 10⁵) to the priority from the age factor. And if we do the same for our largest request in CPUs, 1152, it adds just 32 (1152 divided by the total available CPUs in LUMI, 360 448, multiplied by 10⁴) to the priority due to the size factor. Compared with a fair share of just 0.01 that adds 1000 (0.01×105) to the priority, both the age and the size factors are marginal.

The time in queue of the simulations is a growing issue for those users utilizing highly congested machines. And it is even worse for those executing simulations with vertical workflows, where workflow tasks have to wait until their dependency is met. This paper analyzes for the first time, to the best of our knowledge, a simple but powerful solution to mitigate this issue, which is to aggregate tasks into a single submission to be sent to the HPC platform.

We measured the impact of aggregating tasks by running two simulations under the same conditions with a Slurm simulator, one wrapping and another with all the tasks independently submitted. We used the MONARCH air quality application as a reference for the application. To have both modern job requests and realistic behavior on the usage of the machines, we performed two experiment types: one with synthetic static workloads modeled after a current flagship system, where all the jobs are submitted at the same time, and another with a real dynamic workload, as recorded in a production machine.

We tied this impact with a key scheduling factor employed by system administrators of Slurm: the fair share. This factor quantifies the responsiveness of the machine to the user, and is shared with the group. This factor is normally assigned a high weight, which equates to a larger impact in the job's priority.

Our findings support the thesis that aggregating subsequent tasks reduces queue time. We observed a maximum difference in queue time between unwrapped and wrapped of 7 % of the total runtime of the workflow. If we put this saving in terms of the 10 % to 20 % queue overhead reported by the community, that would be a reduction of the queue time of 35 % to 70 %.

This result supports our thesis that, since we submit fewer jobs, we have fewer tasks of the workflow stuck in the queue, and therefore the overall time-to-solution of the simulation is improved.

As for the fair share, we see how a fair share factor of just 0.5 is enough for the job to be executed immediately in the static experiments. This is not as clear in the dynamic results, but we observe that with a higher fair share, the impact of using wrappers decreases.

Finally, this paper brings to the forefront the issue of queue time in shared HPC platforms, which is particularly acute for the climate community, but it may be shared with other communities running their applications on congested shared HPC machines. We expose the important parts of the inner workings of the scheduling algorithm of the most popular workload manager, Slurm, and relate them to the request in terms of CPUs and wallclock of the workflow. This work is of interest to all those who see the need to optimize their time-to-solution by providing recommendations on how to submit their tasks and the rationale as to why it is beneficial.