We consider a spherical shell of inner radius ri and outer radius ro rotating about the z axis with a constant frequency Ω. Convective motions are driven by a fixed temperature contrast ΔT between the two boundaries. We adopt a dimensionless formulation of the MHD equations using the shell gap d=ro-ri as the reference length scale and the viscous diffusion time d2/ν as a time unit. Velocity is dimensionlessly represented in units ν/d, temperature in units ΔT, and the magnetic field in units ρμλΩ, where μ is the magnetic permeability, ν the kinematic viscosity, and λ the magnetic diffusivity. The dimensionless equations that control the time evolution of the velocity u, the magnetic field B, and the temperature T under the Boussinesq approximation read 1∇⋅u=0,∇⋅B=0,2∂u∂t+u⋅∇u+2Eez×u=-∇p+RaPrgTer+1EPm∇×B×B+∇2u,3∂B∂t=∇×u×B+1Pm∇2B,4∂T∂t+u⋅∇T=1Pr∇2T, where p is the non-hydrostatic pressure, er and ez are the unit vectors along the radial and axial directions, respectively, and g=r/ro is the dimensionless self-gravity. This set of equations is controlled by four dimensionless parameters, the Rayleigh number Ra, the Ekman number E, the magnetic Prandtl number Pm, and the Prandtl number Pr: 5E=νΩd2,Ra=αgod3ΔTνκ,Pr=νκ,Pm=νλ, where α is the thermal expansivity, go is the gravity at the outer boundary, and κ is the thermal diffusivity. Equations ()–() need to be complemented by appropriate boundary conditions for temperature, velocity, and magnetic field as well as an initial state.

To ensure the solenoidal nature of u and B, Eqs. ()–() are solved in the spherical coordinates (r,θ,ϕ) by expanding the velocity and the magnetic fields into poloidal and toroidal potentials following u=∇×∇×Wer+∇×Zer,B=∇×∇×Ger+∇×Her. This formulation yields the six time-dependent scalar quantities W, Z, G, H, T, and p. These six variables are expanded in spherical harmonic functions up to a degree and order ℓmax in the angular directions (θ,ϕ). Since the spherical harmonics are the set of eigenfunctions of the Laplace operator on the unit sphere, they are an especially attractive basis for representing functions in spherical coordinates. They are defined by 6Yℓm(θ,ϕ)=Pℓm(cos⁡θ)eimϕ, where Pℓm represents the normalized associated Legendre polynomials see. Any function f(r,θ,ϕ,t) at the radius r and instant t can be expanded by a truncated spherical harmonic decomposition: 7f(r,θ,ϕ,t)≈∑ℓ=0ℓmax∑m=-ℓℓfℓm(r,t)Yℓm(θ,ϕ). Letting ⋆ denote the complex conjugate, it holds that fℓ,-m(r,t)=fℓm⋆(r,t) for all real functions f. This effectively halves the cost of computing and storing the spherical harmonic coefficients.

Reordering the terms in Eq. () and using the definition of the spherical harmonics from Eq. () we obtain f(r,θ,ϕ,t)≈∑m=-ℓmaxℓmaxfm(r,θ,t)eimϕ8(Fourier transform),fm(r,θ,t)≈∑ℓ=|m|ℓmaxfℓm(r,t)Pℓm(cos⁡θ)9(Legendre transform), which is known as the inverse spherical harmonics transform. By integrating fℓmYℓm over the spherical coordinates using the Gauss–Legendre quadrature we obtain the forward transform: fm(r,θ,t)=12π∫02πf(r,θ,ϕ,t)e-imϕdϕ≈1Nϕ∑k=1Nϕf(r,θ,ϕk,t)e-imϕk10(Fourier transform),fℓm(r,t)=1π∫0πfm(r,θ,ϕ,t)Pℓm(cos⁡θ)sin⁡θdθ≈1Nθ∑j=1Nθwjfm(r,θj,t)Pℓm(cos⁡θj)11(Legendre transform), where θj and wj are respectively the Gauss nodes and weights for the Nθ collocation points in the latitudinal direction, and ϕk=2kπ/Nϕ represents the Nϕ regularly spaced grid points in the azimuthal direction.

The formulation discussed in Eqs. ()–() provides a two-step algorithm for computing the spherical harmonics transforms (hereafter SHTs) corresponding to a Legendre transform and a Fourier transform. The latter can be handled by finely optimized implementations of the fast Fourier transform (FFT for short) such as the FFTW library , which costs O(Nϕlog⁡Nϕ) operations. There is no fast transform for Eq. (), and the complexity of an individual Legendre transform scales as O(Nθ2) operations. In MagIC, we use the open-source library for SHTs and Legendre transforms , which relies on the recurrence relation by . Additionally, in the context of practical numerical implementation, a transposition is required between the two transforms; this will be covered further in Sect. .

The quadrature shown in Eq. () is exact for Nθ>ℓmax , but in practice, we set ℓmax=⌈2Nθ/3⌉ in order to prevent aliasing errors (e.g. ). In fact, we also set Nϕ=2Nθ for the number of longitudinal points, which guarantees isotropic resolution in the equatorial regions .

During the initialization stage, MagIC allows the choice between finite differences or a spectral expansion using Chebyshev polynomials to handle the radial discretization strategy. Finite-difference methods allow the use of faster point-to-point communications, but they also require a larger number of nodal points to ensure a proper convergence of the solution. In this work, we explicitly chose to focus solely on the spectral approach to handle the radial discretization, but we encourage the interested reader to consult for an extensive comparison between several different numerical implementations.

Each spectral coefficient fℓm(r,t) is expanded in truncated Chebyshev series up to the degree Nc-1: 12fℓm(rk,t)≈c∑n=0Nc-1′′fℓmn(t)Cn[x(rk)], where c=2/Nr-1 is a normalization factor and the double quotes mean that the first and last terms need to be multiplied by one-half. In the above expression, Cn(x) is the nth-order first-kind Chebyshev polynomial defined by Cn(xk)=cos⁡[narccos⁡(xk)],xk∈[-1,1]⊂R, with xk=cos⁡(k-1)πNr-1,k∈[1,Nr], the kth nodal point of a Gauss–Lobatto grid defined with Nr collocation points. The discrete radius rk defined between ri and ro is mapped onto xk using the following affine mapping: rk=ro-ri2xk+ro+ri2,k∈[1,Nr]. Conversely, the Chebyshev spectral coefficients of the functions fℓm(r,t) read 13fℓmn(t)≈c∑k=1Nr′′fℓm(rk,t)Cn(xk). With the particular choice of Gauss–Lobatto collocation points, Eqs. () and () can also be efficiently computed using fast discrete cosine transforms (DCTs) of the first kind Sect. 12.4.2.

With the spatial discretization fully specified, we can proceed with the time discretization. For an easier understanding, we will derive the main steps using the equation for the time evolution of the magnetic poloidal potential G as an illustrative example. We refer the interested reader to e.g. or the online documentation of MagIC for the derivation of the other equations.

The equation for G is obtained by considering the radial component of the induction equation (see Eq. ). Using the spherical harmonics decomposition and the Chebyshev collocation method described above we obtain 14ℓ(ℓ+1)rk2c∑n=0Nc-1ddt+1Pmℓ(ℓ+1)rk2Cn(xk)-1PmCn′′(xk)Gℓmn(t)=∫[er⋅∇×(u×B)]Yℓm⋆dS, where Cn′′ is the second radial derivative of the nth Chebyshev polynomial and dS=sin⁡θdθdϕ is the spherical surface element.

Equation () can be rewritten in the following matrix form: 15AℓdGℓm(t)dt-LℓGℓm(t)=Nℓm[u,B], where the matrices Aℓ and Lℓ∈RNr×Nr contain primarily the coefficients of Cn(xk) and Cn′′(xk), are dense, and depend on ℓ but not on m. Equation () forms a set of ordinary differential equations that depend on time and contain a non-linear term, namely Nℓm(t)∈CNr, and a stiff linear diffusion operator.

In order to mitigate the time step constraints associated with an explicit treatment of the diffusion terms, MagIC adopts an implicit–explicit (IMEX) time-stepping approach. Non-linear and Coriolis terms are handled using the explicit part of the time integrator, while the remaining linear terms are treated implicitly. Currently, IMEX multisteps e.g. or IMEX diagonally implicit Runge–Kutta (DIRK; e.g. ) are available.

Let δt denote the time step size. A general k step IMEX multistep method applied to Eq. () reads 16(Aℓ-δtb0ILℓ)Gℓmi+1=∑j=1kajAℓGℓmi+1-j+δt∑j=1kbjENℓmi+1-j+bjILℓGℓmi+1-j, where the exponent notations correspond to the time discretization with ti=t0+iδt and aj, bjE, and bjI corresponds to the weights of the IMEX multistep scheme. In practice, multistep schemes present stability domains that decrease with their order of convergence and require knowledge of past states to continue the time integration. As such, they are not self-starting and need to be initiated with a lower order scheme. MagIC implements DIRK schemes to overcome these limitations, but here we discuss only a simple two-step IMEX scheme. Setting the IMEX weights to a=(1,0), bE=(3/2,-1/2) and bI=(1/2,1/2,0), Eq. () reduces to the popular IMEX scheme assembled from a Crank–Nicolson and a second-order Adams–Bashforth scheme and can simply be written as Aℓ-δt2Lℓ︸MℓδtGℓmi+1=17Aℓ+δt2LℓGℓmi+δt23Nℓmi-Nℓmi-1︸Bℓmi, with Mℓδt∈RNr×Nr and Gℓmi+1,Bℓmi∈CNr respectively defined by the underbraces on the left- and right-hand side of Eq. (). Assuming suitable initial and boundaries conditions, all terms needed to compute Bℓmi are known from the previous iterations. The matrices Mℓδt contain the Chebyshev coefficients and are dense but real-valued and of moderate size (realistic values of Nr range from 33 to 1025). As such, the linear system in Eq. () typically requires O(Nr2) operations to be inverted and can be easily solved using LU decomposition and backward substitution from standard libraries such as LAPACK . From here on, we generalize the notation and use Eq. () when discussing the time stepping for any fields, i.e. not only the magnetic poloidal potential G.

For the sake of simplicity, we discuss only the second order time-stepping scheme described in Sect. , Eq. (). The resulting pseudocode is shown in Algorithm 1. For each time step, the code can be divided into two stages, the “radial loop” and the “ℓ loop”.

The radial loop (lines 2–10). It computes the non-linear terms in grid space (θ,ϕ,r) and thus requires forward and inverse SHTs. The radial levels are completely decoupled in this stage and can thus be efficiently distributed without any need for communication between them. The directions (ℓ,m) and (θ,ϕ) are coupled. Notice that the FFTs on lines 6 and 9 take place on the second, slowest dimension. This requires the Fourier transform to “stride” the data in memory. This is not ideal, but modern FFT libraries are able to efficiently handle strided transforms.

The ℓ loop (lines 13–18). It performs the actual time step. Most of the computational effort goes into solving the linear systems from Eq. () and into updating the right-hand-side Bℓmi. After updating each Bℓmi, the factors of Mℓδt are computed using LAPACK's real-valued LU decomposition dgetrf routines for dense matrices. Next, the solution of each linear system from Eq. () needs to be computed. In practice, all right-hand sides are “packed” in a real matrix Bℓi∈RNr×(2ℓ+2): 18Bℓi=ℜ(Bℓ0i),ℜ(Bℓ1i),…ℜ(Bℓℓi),ℑ(Bℓ0i),ℑ(Bℓ1i),…ℑ(Bℓℓi). where ℜ and ℑ respectively represent the real and imaginary parts of the vector. The system is then solved for all right-hand sides as 19MℓδtX=Bℓi, with a single call to LAPACK's dgetrs. The matrix Bℓi is computed and stored as (r,m,ℓ) for faster memory access during the LAPACK call. Therefore, the intermediate solution X is also obtained as (r,m,ℓ). The final solution Gℓmi+1 is reconstructed from X by reordering the data back into the (ℓ,m,r) format, while adding the real and imaginary parts back together. Also notice that the decomposed factors of Mℓδt are kept for all subsequent time steps and are only recomputed when δt changes.

In this section we discuss how the simulation is distributed across MPI ranks for the 1d-hybrid and the 2d-MPI implementations. For all purposes, MPI Cartesian grid topology is used. The ϕ direction is numerically periodic, but this property is never explicitly used by the underlying algorithms. The remaining directions are non-periodic. We denote by Πr the number of MPI ranks used in the radial direction. Similarly, Πθ represents the number of ranks in the θ or ℓ direction.

Let Nℓ=ℓmax+1 denote the total number of ℓ modes, Nm the number of m modes, and Nℓm the number of (ℓ,m) tuples. We simply use the - superscript to denote the number of local points in some MPI rank. For instance, Nr- denotes the number of local radial points stored in a given rank.

1d-hybrid distribution. In the radial loop, the data distribution for the 1d-hybrid approach follows intuitively from Algorithm 1. The radial shells are distributed evenly among the MPI ranks, ideally Nr-=Nr/Πr for each rank. During the computation of the non-linear terms, each rank stores Nϕ×Nθ×Nr- points. During the computation of the miscellaneous terms, each rank stores Nℓm×Nr- points.

This changes for the execution of the ℓ loop. For efficiently computing line 17 of Algorithm 1, all Nr radial points must be local. This is achieved by the so-called ℓ transposition, which changes the data distribution to Nr×Nℓm- local points. Special care is needed when choosing the distribution of the Nℓm- points. The computational effort per ℓ mode is given by the LU decomposition of the matrix and the solve step for ℓ+1 right-hand-side vectors for this matrix. To balance the computation among the Πr processes, we distribute the ℓ modes in a “snake-like” fashion. Starting with the largest ℓ, which has the highest computational effort, we distribute the ℓ modes over the processors until Πr is reached. The next ℓ is also placed on this last processor, and then we reverse the order until reaching the first processor again. The procedure is repeated until all modes have been distributed. See Fig.  for an illustration with Nℓ=21 and Πr=6. This snake-like distribution guarantees that the largest possible imbalance between two MPI ranks is Πr-1 tuples. Furthermore, each matrix Bℓi can be fully stored locally, which is not the case for the 2d-MPI code, as we discuss below. At the end of the ℓ loop, another ℓ transposition takes place, converting the data layout back to Nℓm×Nr- local points.

2d-MPI distribution. Like in the 1d-hybrid implementation, the radial points are split evenly between the Πr ranks in the radial loop. Additionally, the Nθ points are distributed evenly and contiguously between the Πθ ranks (see Fig. a). At line 7, each rank is responsible for the computation of Nθ-×Nϕ×Nr- points. After the FFT at line 9 of Algorithm 1, Nθ-×Nm×Nr- points are stored locally, but the computation of the Legendre transform at line 10 requires all θ angles for a given spherical harmonic order m to be available locally. This is guaranteed by the so-called θ transposition, which gathers the values for all Nθ θ angles while distributing the m modes, resulting in Nθ×Nm-×Nr- local points.

The distribution of the m modes is more involved than the distribution of the θ angles because there are (Nℓ-m) different spherical harmonic degrees ℓ for each order m. Once again, we employ the snake-like distribution depicted in Fig. , which guarantees that the largest imbalance between any two MPI ranks is Πθ-1 tuples. See Fig. c–d for a graphical representation of this step.

Just like in the 1d-hybrid implementation, an ℓ transposition is required before the ℓ loop in order to locally gather all Nr radial points. This once again involves a complex redistribution of data points. Because of Eq. (), we would like to distribute only the ℓ modes while keeping all radial points and m modes local. This would incur an unfeasible volume of data transfer.

We opted for a compromise that exploits a special regularity of the current data structure and leaves the m modes local. Only the Πθ ranks having a common m mode then need to communicate in the radial direction. Figure d–e visualize how the respective communication works. Only the magenta and red ranks need to share their radial information since they represent the same m modes. They then also have the same number of ℓ modes and can thus split the number of modes. The snake-like distribution leads to a good balancing for both ranks. The resulting distribution is visualized in Fig. e.

The figure also illustrates a drawback of this distribution: the m modes for one l mode can be found on different ranks. This means that the Bℓi matrices have their columns distributed between MPI ranks and that the time-stepping matrices Mℓδt and their respective decomposition need to be duplicated in Πθ ranks. However, our numerical experiments show that the communication cost far outweighs the cost associated with solving Eq. (), thus justifying our choice. This will be further discussed in Sect. . Finally, we would like to mention that a similar data layout was employed by in their two-dimensional MPI decomposition of their pseudo-spectral code in spherical geometry.

The spherical harmonics transform (SHT) takes a substantial portion of the runtime. MagIC relies heavily on SHTns , a highly optimized library dedicated to computing the spherical harmonics transform using OpenMP, advanced SIMD vectorization, and cache-hitting strategies . SHTns has been developed to compute the steps synthesized in lines 8–10 and lines 4–6: that is, independent of the radial level r. It can handle scalar fields and is also optimized for general vector fields and vector fields with a poloidal–toroidal decomposition. SHTns is written in C but has a Fortran interface with a dedicated data layout for MagIC. Although SHTns is flexible and modular, it does not have an MPI-distributed interface. However, it offers an interface for computing the Legendre transform (lines 5 and 10 of Algorithm 1) only for an individual (m,r) pair, which is the use case in our 2d-MPI implementation. The transposition and FFT computations are then handled separately by MagIC.

Next we describe how both the 1d-hybrid and 2d-MPI version of the code use SHTns for computing the SHT.

1d-hybrid radial loop. The spherical harmonics transforms are delegated directly to SHTns (see the pseudocode in Algorithm 2). For instance, for the temperature scalar field on the numerical grid, T(θ,ϕ,rk), a single call to spat_to_SH returns the fully transformed Tℓm(rk) for a given radial level rk. This is only possible because all (θ,ϕ) points are local in the 1d-hybrid version of the code. No explicit optimization or multithreading is required, as SHTns handles this internally. The number of vector transforms needed per radial loop depends on the set of equations. Dynamo models under the Boussinesq approximation of the Navier–Stokes equations considered here (Eqs. –) involve four vector transforms and one scalar transform for the backward SHTs and three vector transforms for the forward SHTs.

2d-MPI radial loop. The pseudocode for our implementation is shown in Algorithm 3. Since shtns_ml is called for single modes only, the implementation cannot benefit from the efficient internal OpenMP parallelization of SHTns. The computation time for only a single mode is too small to justify the initialization and termination of a multithreading context. In addition, the larger number of SHTns calls causes some overhead. Some speed-up could possibly be gained with an OpenMP parallelization of the SHTns calls (lines 3–5 in Algorithm 3) or directly at the radial loop level (line 1 in Algorithm 3). Some preliminary tests showed promising results for the former. However, here we shall continue discussing only the pure MPI version.

The so-called θ transposition happens in line 8 of Algorithm 3 and involves MPI communication. Pointers to the input fields hi and output fields fi are ordered in a queue. Upon calling finish_queue, each hi is effectively transposed. After receiving the data, a complex-to-real FFT (e.g. using ) is performed and saved into the output fields fi. The queue has a limited size. Once the limit is reached, finish_queue is called, immediately triggering the data transfer and the FFT computation. Additionally, upon exiting the loop, finish_queue is called one last time to treat any remaining field in the queue.

The θ transposition is implemented using the following algorithms.

P2P: each θ rank performs one call to mpi_isend and mpi_irecv for each other θ rank (i.e. Πθ-1) for all queued fields, followed by an mpi_waitall call. MPI types are used to stride the data when needed. The advantage of this algorithm is that it does not require any particular packing and/or reordering of the data.

Two parameters need to be determined, namely which MPI algorithm to use and the length of the queue. The choice of the MPI algorithm depends on the MPI implementation, the hardware, and the number of resources. We compare the performance of each variant in Sect. .

As for the length of the θ-transposition queue, several factors must be taken into account. Longer queues allow more scalar fields to be packed per call, thus allowing larger message size and decreasing the impact of the latencies of the MPI calls. On the other hand, longer queues require larger send and receive buffers. The benefit of having larger message sizes may be not relevant for all configurations, especially if all θ ranks are located within the same NUMA domain (for instance, the Intel implementation of mpi_alltoallv automatically uses shared-memory operations in this regime). Furthermore, larger queue sizes do not guarantee full usage of the buffers. We discuss this in more detail in Sect. .

As discussed in Sect. , the data need to be redistributed with the so-called ℓ transposition prior to the execution of this part of the code. The distribution of the ℓ modes and their respective m modes to the MPI processes has already been discussed in Sect. .

Most of the wall time of the ℓ loop is spent in the ℓ transposition and in the backward substitution for solving the linear systems from Eq. (). Depending on how often δt changes, the cost for computing the LU decomposition for each dense matrix Mℓδt might become a bottleneck, but in the scenario investigated during our benchmarks, this was not the case. Nevertheless, a frequent change in δt indicates that the algorithm should be run with a generally smaller time step.

Next we describe the ℓ-loop implementations of both MagIC versions.

1d-hybrid ℓ loop. The pseudocode is shown in Algorithm 4. The ℓ transposition itself takes place in lines 1 and 20 of Algorithm 4. Inside one MPI rank, the work is parallelized with OpenMP. We have a two-level loop structure, with an outer loop over the different ℓ degrees on the MPI rank and for each ℓ a loop over all possible m≤ℓ. A single OpenMP parallelization of the outer loop would directly compete with the MPI parallelization and would also lead to a strong load imbalance as the different harmonic degrees have different numbers of m. To avoid nested OpenMP regions (for the outer and inner loop) we use the OpenMP tasking approach instead to balance the work and to combine the parallelism of the outer and inner loop. We create an OpenMP task for each outer loop iteration which contains the LU decomposition of the matrix and a task creation loop for the inner m loop. To minimize the overhead for OpenMP task creation, we use chunks of several m together in one task.

2d-MPI ℓ loop. The pseudocode for the ℓ loop in the sequential and the 2d-MPI implementations (see Algorithm 5) differs only in the ℓ transpositions and the number of ℓ and m modes stored locally (which consequently affects the number of linear systems and right-hand sides in each rank). The distribution of the (ℓ,m) tuples has already been discussed in Sect. . While the difference in the number of modes has an impact on the computation of the solution of Eq. (), this pales in comparison with the time spent in communication.

In both the 1d-hybrid and 2d-MPI implementations the ℓ transposition might be performed using the P2P, A2AV, and A2AW algorithms discussed previously. Additionally, the following algorithm has been implemented.

A2AP: a single call to mpi_alltoall per scalar field is performed with padding to accommodate the largest message. No MPI types are needed, but the data need to be reordered in a buffer. Due to the intrinsic imbalance in the distribution of the data, considerably larger buffers are required. Some MPI libraries implement faster algorithms for mpi_alltoall, which can in some cases be advantageous.

The performance of this code section is of special interest. This is because the θ transposition is required only for the 2d-MPI implementation, thus adding unavoidable extra cost for handling the two-dimensional distribution of the data.

For the tests we fixed the size of a scalar field in grid space to 240 KiB per rank per radial level. This can be achieved by using the formula Nθ=4810Πθ to determine the number of latitudinal points.

We expect the performance of the θ transposition to strongly depend on the distribution of the θ ranks across the computing nodes. To explore this behaviour, we defined three communication regimes with the following parameters.

intranuma: θ communication is constrained within a single NUMA domain; Πθ=10,Πr=4, Nθ=480, and Nr=128.

We measure the performance of the θ transposition in these three different regimes and determine for which algorithm (A2AV, P2P, or A2AW, discussed in Sect. ) and for which queue length (for Algorithm 3) they perform best. Notice that larger queue lengths do not guarantee full usage of the larger buffers. As an example, suppose that qmax is the queue length and that q fields are queued for transposition. If three new fields are added to the queue (i.e. a vector field) but the q+3>qmax, the algorithm transposes the first q fields, clears the queue, and then adds the three fields into a new queue. This could be improved to enforce full buffer usage in future versions, but our benchmarks demonstrate that the benefits would be marginal. Table  shows the effective usage of the transposition buffers for queue lengths from 3 to 13, as well as the “queue-free” version denoted by queue length 1.

Figure  shows the times for 50 time steps for the three different communication regimes. The inferior performance of the A2AV implementation is attributed to the time spent reordering the send and receive buffers. Both A2AW and P2P use the same MPI types to avoid this reordering and therefore perform very similar. Concerning the queue length, the intranuma and intranode regimes perform best with smaller queues, while the opposite is true for the internodes regime.

Since the A2AW implementation for the θ transposition seems to perform best, we stick to this scheme for the following test wherein we address the impact of the queue length.

In this subsection we continue the comparison between intranuma, internuma, and internode regimes but now bring the whole spherical harmonics transform into perspective. Since this constitutes one of the largest portions of the computation, it is particularly interesting to determine the behaviour in the different communication regimes.

Figure shows the θ-transposition time as well as the computation time spent inside the FFT and Legendre transform calls. Although the computation time itself should not depend on the queue length, Fig.  shows a small but consistent influence. We assume that this is linked to the change in data access pattern caused by queuing and buffering data for the θ transposition.

Even for the intranuma case shown in Fig. a, where a queue length of 3 provides a small performance boost for the θ transposition, the performance of the SHT is optimal for a queue length of 1. In the following test we therefore fix the queue length to 1 for intranuma and intranode regimes and to 13 for the internode regime.

In this subsection we compare the performance of the four different ℓ-transposition implementations for the 2d-MPI version. We would like to recall that the 1d-hybrid implementation uses an auto-tuning routine which automatically selects the best algorithm during the initialization. Moreover, the main difference between 1d-hybrid and 2d-MPI implementations in the ℓ loop is the distribution of the (ℓ,m) tuples, as already discussed in Sect. . The communication pattern therefore differs significantly for the two implementations.

The problem we chose for this and subsequent subsections is the dynamo benchmark with Nr=301 and Nθ=960. We compute 100 time steps for several values of Πr and Πθ, as well as for core counts ranging from 120 to 24 000 (3 to 600 nodes, respectively). For the 1d-hybrid implementation, we use 10 OpenMP threads per MPI task for tests with up to 1000 cores (25 nodes) and 20 threads for larger runs (this configuration resulted in the best total time). The 1d-hybrid version of the code can use at most 6000 cores (150 nodes) with one radial point per socket (Πr=300 and 20 threads per MPI task).

For the 2d-MPI scheme, we fix the θ-transposition implementation to A2AW (as discussed in Sect. ) and the queue lengths to 1 for Πθ=10, 20, or 40 and 13 otherwise. This new version of MagIC admits more combinations of Πθ and Πr and allows the use 24 000 cores (600 nodes). For some core counts, multiple values of Πθ and Πr were allowed. For example, for 6000 cores one may choose Πθ=20 or Πθ=40 with Πr=300 or Πr=150. We tested both and kept the configuration with the best total time. For the reader's convenience, all plots show the value of Πθ for the 2d-MPI version at the top horizontal axis. We also set the experiments such that Nr-1 is divisible by Πr and assign the last radial point to the last radial MPI rank. This is currently a limitation of MagIC. In principle, this should cause imbalance in some cases; e.g. with Πr=300 the last radial MPI rank will store and solve twice the number of radial levels as the other ranks. However, the last radial MPI rank receives fewer (ℓ,m) tuples, which diminishes the impact of the imbalance. Since both codes suffer from the same imbalance, we opt for simplifying the analysis to the average runtime only.

In Fig.  we can see that, except for the A2AP variant, the 2d-MPI implementation is always significantly faster than the 1d-hybrid implementation. This is a direct consequence of the communication-avoiding pattern of the 2d-MPI distribution.

The 1d-hybrid version shows a good strong scalability, which, however, starts to degrade after 1000 cores (25 nodes). At this core count the 1d-hybrid version transitions from 10 to 20 threads. OpenMP efficiency is typically difficult to maintain for large thread counts, and this is likely the reason for the performance degradation.

The 2d-MPI implementation, especially for the A2AW variant, scales remarkably well up to 6000 cores (150 nodes), at which point the performance starts to degrade. However, the losses remain acceptable up to the largest number of cores we could test: 24 000. The A2AV variant performs similarly to A2AW, and beyond 12 000 cores all variants show nearly identical timings. This test suggests preferring A2AW, which will be kept in the following. However, as the behaviour may be different on other architectures and for other MPI libraries, an auto-tuning routine that selects the fastest option in the initialization phase of a simulation seems a good idea for the future.

In this subsection we take a deeper look into the different parts of the ℓ loop. We are especially interested in the impact of the 2d-MPI data distribution on the LU decomposition and on the backward substitution. For these tests we use the same setting as in the previous subsection (100 time steps, Nr=301, and Nθ=960) and stick to the A2AW implementation of the transpositions.

In Fig.  we compare transposition time, LU decomposition time, and the backward substitution time (referring to Eq. ) for the 1d-hybrid and the 2d-MPI implementations. In addition, the total times for the ℓ loop are shown, which also include the updating of time derivatives and related operations.

In both the 1d-hybrid and 2d-MPI implementations, the performance is dominated by the transposition time. Here the 2d-MPI clearly outperforms the 1d-hybrid implementation because of the reduced communication, as discussed in Sect. .

The timing results in Fig.  demonstrate that decomposition and backward substitution for the linear systems do not scale well with Πθ (upper horizontal axis). As explained in Sect. , the data are distributed in the direction of the m modes, but the time-stepping matrices, which need to be LU-decomposed, are independent of m.

For a fixed Πr, the decomposition time is thus expected to remain independent of Πθ. Likewise, the backward substitution time should just mildly be impacted by larger Πθ. This can be clearly seen in Fig. : 120 and 240 cores have Πr=12, with Πθ=10 and 20, respectively. Their decomposition time is roughly the same, with a slightly faster backward substitution time for 240 cores. The same can be observed for 200 and 400 cores as well as 3000 and 6000 cores. There is a spike in the decomposition time for the 1d-hybrid implementation at 1200 cores.

However, the performance gain due to the communication-avoiding distribution of the 2d-MPI implementation far outweighs these shortcomings. We would like to highlight the fact that it is only possible to reduce the communication volume due to the particular distribution pattern shown in Fig. ; i.e. each θ rank communicates with Πr ranks in the radial direction. The communication-avoiding scheme discussed here is not possible for the 1d-hybrid implementation, since one of these directions is missing. This specific feature of the 2d-MPI version is essential to enable a better overall performance for large cores count, as we shall discuss in the next subsection.

Finally we discuss the strong scaling of the MagIC time step, i.e. of the main application without initialization, finalization, and diagnostics. As usual, we define the parallel efficiency as 20ϵ=tref.×nref.t×n, where t is the main application time in seconds and n is the number of cores used. We use the test with nref.=120 as a reference.

Following , we consider ϵ>0.6 to be a sufficiently good efficiency. Additionally, we define the “parallel cross-efficiency” ϵ* that compares the 2d-MPI timing to the 1d-hybrid reference time tref.=405.6s for nref.=120 as a baseline. The timings and parallel efficiency, along with other information about the tests, are listed in Table . The graphical visualization of these values (also discriminating the time for the ℓ loop and radial loop) is shown in Fig. . Since we use the same problem size and settings already discussed in Sect.  and the curves for the ℓ loop are the same as in Fig. .

In Fig.  it is interesting to notice that the performance of the 1d-hybrid implementation is dominated by the ℓ-loop performance, whereas the performance of the 2d-MPI implementation is dominated by the performance of the radial loop. To some degree this is expected because the radial loop of the 2d-MPI implementation requires the θ transposition in addition to the SHT.

The radial loop for the 1d-hybrid code scales well up to 6000 cores, but the inferior scalability of the ℓ loop already leads to a noticeable degradation beyond 1200 cores in Fig. . This is the cause of the decrease in parallel efficiency ϵ apparent in Table  for 2000 or more cores.

The 2d-MPI implementation starts suffering from a small loss of scalability of the ℓ loop at 9000 cores but remains at an acceptable level up to 24 000 cores. The radial loop scales remarkably well up to 12 000 cores but shows degraded performance at 18 000 and 24 000 cores. This can be explained by the problem size being too small for such numbers of cores, which is reflected by Nr-=1 and Nℓ-≈22 for 9000 cores. Both the ℓ and the radial loop contribute to the loss of overall performance at the highest core counts apparent in Fig. .

The parallel cross-efficiency (see Table  and Fig. ) shows that the new 2d-MPI version is about 20 % slower than the 1d-hybrid implementation for core counts of n<2000. However, the 2d-MPI variant scales better for n≥2000 with ϵ=0.7 for up to n=12000. The 1d-hybrid implementation, on the other hand, reaches ϵ=0.63 for 6000 cores. The 2d-MPI is as fast as the 1d-hybrid implementation at n=3000 but already 10 % faster at n=6000.

The benefits offered by the 2d-MPI implementation can be illustrated with a simple example. Instead of performing a simulation with the 1d-hybrid implementation on 6000 cores a scientist could opt for using the 2d-MPI implementation on 12 000 cores and obtain the solution in only 56 % of the runtime. In other words, by investing only 12.5 % more CPU hours, the 2d-MPI implementation arrives at the same solution in half the time. Furthermore, using the 2d-MPI implementation allows the user to allocate more computing nodes and thus gives the user access to more distributed memory.

The parallel cross-efficiency also gives an idea of how “far” the parallel efficiency of the 2d-MPI is from the 1d-hybrid implementation. There is a significant gap for 120 to 1200 cores, but the gap closes at about 2000 cores, e.g. ϵ=0.86 for the 1d-hybrid and ϵ*=0.82. This trend continues until 6000 cores, when the parallel cross-efficiency overtakes the parallel efficiency of the 1d-hybrid implementation.

Finally, the main characteristics measured on our benchmark platform are expected to be transferable to other flavours of HPC platforms based on contemporary x86_64 CPUs (e.g. AMD EPYC) and high-performance interconnects (e.g. Mellanox Infiniband), provided the platforms are sufficiently optimized for SHTns and MPI libraries. In general, for good performance of the 1d-hybrid version it is mandatory to establish the optimal number of threads per MPI rank and to respect the placement of the threads within the NUMA domains. The new 2d-MPI version brings the advantage of being less affected by the actual node topology, but the performance of the application is sensitive to the MPI library optimization of routines such as mpi_alltoallw for shared-memory environments.

The parallel cross-efficiency in Table  shows that the performance of both implementations is rather close for core counts larger than 2000. This rightfully raises the question of whether this small gap could be bridged. Furthermore, it has already been established in previous sections that the main bottleneck of the 2d-MPI implementation is the radial loop, but the θ transposition and the computation time (i.e. mostly the SHT) of the radial loop have not yet been thoroughly compared. The goal of this subsection is to provide this comparison and discuss which portions of the code could be optimized and what impact this could have on the 2d-MPI implementation.

For the 1d-hybrid code, let tt and tr respectively denote the full time of the main application and the radial loop only, both given in seconds. Analogously, for the 2d-MPI code, let Tt denote the time of the full main application and Tr+Tθ denote the time of the radial loop only, where Tr is the computation time only and Tθ is the θ-transposition time only (including communication and necessary buffer copies, if any). Table  shows tr, Tr, and Tθ for the same experiment from 120 to 6000 cores. Both tt and Tt have the same values shown in the “time” columns of Table .

Table  shows that the θ-transposition times comprises on average 17.5 % of the radial loop (except for 6000 cores, for which it takes circa 28 % of the time of the radial loop). The runtime of θ transposition can be further optimized e.g. by introducing complex strategies such as communication–computation overlap. However, from Table  it is evident that the main bottleneck is indeed the computation time Tr.

The next natural question is how the computation times of the radial loop of both implementations compare with each other. For that we use the ratio 21ρ=Trtr. Ideally, ρ would be always close to 1, in which case the computational efficiency of the radial loop of both implementations would be identical. In practice, distributing the data between more MPI ranks means that each rank has less local data, which makes cache-hitting, vectorization, and other fine-grained optimization techniques less efficient. In other words, even a finely optimized implementation of the 2d-MPI data distribution algorithm is likely to have ρ>1.

The column ρ from Table  shows that the performance gap is far from 1 in most cases and is inversely proportional to the number of cores. This means that bridging the large gap in performance for the lower core counts might be unfeasible, but for larger core counts it is attainable.

We now attempt to determine how much improvement in the 2d-MPI implementation is required for its main application runtime (including all transposition times) to match the runtime of the 1d-hybrid code. In practice, several portions of the 2d-MPI code could benefit from cache-hitting strategies, fine-tuned vectorization, and computation–communication overlap, amongst others. We will now discuss a much simpler scenario in which only the computation time of the radial loop can be improved from Tr to κTr. Subtracting Tr from the total time Tt and adding the “new” runtime κTr, we obtain the following equation: 22Tt-Tr+κTr=tt,κ=tt-TtTr+1. which gives us the needed improvement factor κ. The values are shown in Table .

Table  shows that, for low core counts, improvements in Tr alone could not bridge the gap in the performance between 2d-MPI and the 1d-hybrid implementations. From 120 to 1000 cores we have tr>κTr, meaning that the 2d-MPI implementation would have to compute the radial loop faster than the already highly optimized 1d-hybrid code for both codes to perform the same.

The last three entries in Table  show a much more realistic improvement factor. An improvement of 10 % (7 %) in Tr would suffice to bridge the gap between the main application time for 2000 cores (3000 cores). For 6000 cores, Table  shows that Tr could be 31 % slower and both codes would still be equivalent. This is due to the poor scalability of the ℓ transposition of the hybrid code, which gives the 2d-MPI some “room” for performance loss in other portions of the code.