Lithospheric deformation and mantle flow associated with geodynamic processes are described by the Stokes equations for incompressible viscous creeping flow (Eqs. 1 and 2). We incorporated the thermal (Ra) and compositional (Rb) Rayleigh numbers (Eq. 3) into the buoyancy term (e.g., van Keken et al., 1997; Tan et al., 2011). Equations (1) and (4) assume a general form, with each benchmark activating only the terms required by the physical setting. Thermal-buoyancy cases set Rb=0, while compositional-buoyancy cases set Ra=0 in the absence of a temperature field. The presence of a temperature field determines whether Eq. (4) enters the solution. Each calculation adopts one of the two settings alone, keeping the thermal and compositional drivers separate. 1∇⋅τ-∇p+RaT-∑iRbiCiz^=0where  τ=2ηε˙  and  ε˙=12(∇u+(∇u)T)2∇⋅u=03Ra=αρ0gΔTh3κη0  and  Rbi=Δρigh3κη0 p, u, τ, η, z^, and ε˙ denote pressure, velocity vector, the deviatoric stress tensor, shear viscosity, the upward unit vector, and the deviatoric strain-rate tensor, respectively. T and C refer to the temperature and composition fields, respectively. κ, η0, h, and g are thermal diffusivity, reference viscosity, the depth of the domain, and gravitational acceleration, respectively. α, ρ0, ΔT, and Δρi represent the thermal expansivity, reference density, temperature contrast between the bottom and top, and the density contrast of the ith composition relative to ρ0, respectively. For the Stokes system, we employ two mixed finite-element pairs, the Taylor–Hood element (P2×P1) and the MINI element (P1+×P1). Both discretizations use the same P1 pressure space, while the P1+ velocity space enriches P1 with bubble functions. The pair P1+×P1 is commonly referred to as the MINI element in the geodynamics literature (Thieulot and Bangerth, 2025). The MINI element was adopted to reduce the total number of DOFs in high-resolution and 3D simulations. The thermal evolution of the system is governed by the advection-diffusion equation (Eq. 4). 4∂T∂t+u⋅∇T=∇⋅(∇T)

The temporal evolution of the multi-composition field depends only on advection through the velocity field. We adopted a level-set method to reduce oscillations and numerical diffusion of the compositional field under advection-dominated conditions (Hillebrand et al., 2014; Wú et al., 2023). The advection of the level-set function ϕ governs the evolution of the composition field (Eq. 5). The level-set function is reinitialized to a signed-distance function satisfying |∇ϕ|=1 by solving the reinitialization equation (Eq. 6) using second-order ENO spatial discretization, a Godunov Hamiltonian, and RK2 pseudo-time integration. 5∂ϕ∂t+u⋅∇ϕ=06∂ϕ∂τ+S(ϕ0)(|∇ϕ|-1)=0,whereS(ϕ0)=ϕ0ϕ02+ε2 S, ϕ0, and ε denote the smoothed sign function, the level-set function before reinitialization, and the smoothing parameter, respectively, with ε=1.5min⁡(1Nx,1Ny). Nx and Ny are the numbers of grid cells in the x and y directions, respectively. The level-set function ϕ is a continuous scalar field with the interface defined by ϕ=0. The different material domains are distinguished by ϕ<0 and ϕ≥0. Using the composition field obtained via the level-set functions, density and the logarithm of viscosity were linearly interpolated. To define n compositions, n-1 level-set functions are required. We employed one to four level-set functions (one for Rayleigh–Taylor instability and four for the subduction model).

Discretization of the incompressible Stokes equations yields a 2×2 block saddle-point system (Eq. 7). 7KGGT0up=f0

The velocity stiffness matrix K is symmetric and positive definite (Davies et al., 2011). G and GT represent the gradient and divergence operators for the pressure and velocity fields, respectively (Silvester and Wathen, 1994; Benzi et al., 2005; May and Moresi, 2008). f denotes the body force vector. In geodynamic applications, the viscosity contrast ranges from 103 to 106, which leads to an ill-conditioned system and slow convergence of iterative solvers. Furthermore, using a direct solver (e.g., MUMPS) on the full block system causes excessive memory consumption for large-scale problems. Therefore, iterative methods are essential for large-scale problems. One effective approach is to exploit the Schur complement structure of the saddle-point system.

With Schur complement reduction, the Stokes system is transformed into a block upper triangular form (Wathen and Silvester, 1993; Eq. 8). 8KG0Sup=fh^,S=GTK-1G,h^=GTK-1f

To solve the equivalent system GTK-1Gp=GTK-1f, we apply a PCG iteration with the inverse viscosity mass matrix as preconditioner (Eqs. 9–14; Jang et al., 2025).

Initialization: 9p0=0,Ku0=f-Gp0(initial solve)10〈q0,r〉=〈∇⋅u0,r〉,1ηw0,r=〈q0,r〉,d0=-w0(initial residual and precond.)

For k=0,1,2,…: 11Khk=Gdk,αk=〈qk,wk〉〈∇dk,hk〉(inner solve and step size)12pk+1=pk+αkdk,uk+1=uk-αkhk(vector update)13〈qk+1,r〉=〈∇⋅uk+1,r〉,1ηwk+1,r=〈qk+1,r〉(divergence and precond.)14βk=〈qk+1,wk+1〉〈qk,wk〉,dk+1=-wk+1+βkdk(CG update)

The pressure is updated along the conjugate direction dk, while the velocity is updated using the corresponding velocity-space vector hk. The iteration stops when the divergence error (∥∇⋅u∥/∥u∥) of velocity is lower than the divergence tolerance. The algorithm (Eqs. 9–14) requires only a fixed number of vectors in the outer PCG iteration: approximately four in the pressure space (qk, dk, pk, wk) and two in the velocity space (hk, uk). The inner velocity subproblem is solved with GMRES, requiring a separate Krylov basis. However, the symmetric positive definiteness of the velocity stiffness matrix K also permits CG as an alternative inner solver. CG uses a short recurrence with only a few stored vectors, offering a potential reduction in Krylov-vector memory. A further option applies a fixed number of BoomerAMG V-cycles directly as a spectrally equivalent approximation of K-1 within the outer PCG iteration, removing the inner Krylov solver. Outside the velocity subproblem solve (Eq. 11), the remaining operations include vector inner products, scalar–vector multiplications, and vector additions (Eqs. 12 and 14). Memory bandwidth limits the speed of the three operations, which map directly onto GPU architectures. The dominant cost per iteration is the AMG-preconditioned GMRES solve of the velocity subproblem Khk=Gdk (Eq. 11), which involves repeated sparse matrix-vector products and preconditioner applications. The L2-projection steps that apply the mass matrix and the inverse viscosity mass matrix (Eqs. 10 and 13) also offer substantial potential for GPU acceleration. Evaluating the GPU performance of the velocity subproblem solve and the L2-projection is therefore essential for assessing the overall solver efficiency.

To evaluate the computational efficiency (parallel performance) and accuracy (regarding incompressibility) of solving 2D and 3D geodynamic problems, we established independent CPU-based and GPU-based computing environments. In both environments, we utilised FEniCS (ver. 2019.2.0.64), NumPy (ver. 1.23.5), and PETSc (ver. 3.15.5 on CPU and ver. 3.23.2 on GPU; Balay et al., 2025). Although early development of PETSc focused on MPI-based CPU parallel computing, CUDA-based GPU support was available in PETSc 3.4 (2013), whose documentation included the AIJCUSPARSE matrix type for sparse-matrix operations on NVIDIA GPUs using the cuSPARSE library. PETSc 3.9 (2018) made CUDA-based GPU backends available in the release version, rather than only in the development branch. PETSc 3.12 (2019) further refined the VecCUDA interface and introduced in-place vector-array access that avoids unnecessary data transfers from GPU to CPU. PETSc therefore provides GPU-resident vector and sparse-matrix operations required for efficient iterative linear solvers.

GPU acceleration is extendable to full Krylov iteration cycles, covering solvers such as CG and GMRES. Within the finite element framework of FEniCS, the PETSc backend provides access to the same Krylov subspace solvers (CG, GMRES, MINRES, and FGMRES) and a broad suite of preconditioners, ranging from incomplete LU to geometric or algebraic multigrid, in both GPU and CPU environments. A previous study (Zheng et al., 2013) reported up to a 4.8-fold speedup for a finite-difference Stokes problem on a single GPU (NVIDIA Tesla C2070) with a customised GMG solver implemented in MATLAB and CUDA, compared with an Intel i7 workstation (4 cores and 8 threads) running MATLAB with C extensions. Nevertheless, attempts to integrate GPU acceleration into well-established FEM software for high-performance geodynamic simulations remain limited.

In the pressure correction step of the iterative algorithm, an L2-projection using the mass matrix repeatedly projects the divergence of the velocity field onto the P1 space. With FEniCS GPU acceleration, matrix assembly is performed on CPU cores tied to MPI ranks mapped to GPUs. Although GPU-side matrix assembly has already been developed (Cecka et al., 2011; Markall et al., 2013), performing CPU assembly remains a limitation of our code implemented in FEniCS 2019. Under FEniCS GPU acceleration, sparse matrix assembly runs on the CPU using 1 to 16 cores. A matrix assembled on a single CPU core is solved on 1 GPU. Matrices assembled across multiple CPU cores are solved on 1 or 2 GPUs via NVIDIA Multi-Process Service (MPS). Reassembling the mass matrix at each iteration causes repeated GPU-CPU transfers and synchronization, increasing overhead and overall execution time. To minimise these bottlenecks in the GPU implementation, we adopted a pre-assembly strategy in which the P1 mass matrix, together with differential operators (gradient and divergence), is assembled only once, then kept resident in GPU memory for reuse throughout subsequent iterations. The memory required for the pre-assembled matrices was 510 MB for 3.4×106 DOFs and 1.3 GB for 8.9×106 DOFs (Table S2). For a fair comparison between CPU and GPU runs, the same pre-assembly strategy using system memory was also applied to the CPU implementation.

We employed the GMRES Krylov subspace solver to solve the system Khk=Gdk (Eq. 11) within the pressure Schur complement iteration. To accelerate GMRES convergence, the BoomerAMG algebraic multigrid (AMG) preconditioner from the HYPRE library was applied, interfaced through PETSc (Falgout et al., 2006). In different computing environments (i.e., CPU and GPU platforms), optimal convergence efficiency and computational performance depend on careful tuning of the BoomerAMG preconditioner hyperparameters, including coarsening schemes (e.g., Falgout, Parallel Modified Independent Set (PMIS), and Hybrid Modified Independent Set (HMIS)), smoothers (e.g., Chebyshev, Gauss-Seidel, Jacobi, symmetric SOR, and symmetric Jacobi), and interpolation options (e.g., L-shaped, Extended, and Direct). We optimised the BoomerAMG preconditioner settings by considering architectural differences between a 32-core CPU and a GPU with thousands of arithmetic units.

In the GPU environment, we found configurations that exploit massive parallelism while minimising communication and data dependencies. For coarsening, PMIS and HMIS reduced the computational time for coarse-grid construction (De Sterck et al., 2006; Alber and Olson, 2007). To enhance parallel implementation, we adopted the scaled Jacobi method and Extended+i for smoothing and interpolation, respectively (Offermans et al., 2020; Park et al., 2015). A V-cycle served as the multigrid schedule. In multi-core CPU environments, we selected solver configurations that provide strong per-iteration error reduction. We adopted the Falgout algorithm, a standard HYPRE BoomerAMG coarsening strategy, as the default option on the CPU (Falgout et al., 2006; Henson and Yang, 2002). To ensure fairness relative to GPU runs, the HMIS algorithm was tested as an additional coarsening option. Smoothing employed symmetric SOR and Jacobi smoothers, which provide substantial error reduction within a single iteration (Brandt, 1977; MacLachlan and Oosterlee, 2011). Classical interpolation, highly compatible with Falgout coarsening, served as the interpolation scheme (Brannick et al., 2018). As in the GPU configuration, a V-cycle was used for the multigrid schedule.

We evaluated the accuracy, execution time, and VRAM usage of the preconditioned conjugate-gradient solver on the GPU using manufactured solutions, the SolCx viscosity-contrast benchmark (Zhong, 1996; Thieulot and Bangerth, 2022), the sticky-air free-surface benchmark (Crameri et al., 2012; Hillebrand et al., 2014), 2D Rayleigh–Taylor instability tests (van Keken et al., 1997; Maljaars et al., 2021), and the thermo-mechanical convection benchmark of Blankenbach et al. (1989). In 3D runs, stable execution was achieved on a 100×40×100 grid under P1+×P1 element discretization on a single 48 GB GPU, despite the limited VRAM capacity of the GPU. Memory distribution efficiency in a two-GPU system was assessed for memory-demanding cases, including high-resolution 3D runs approaching 100×80×100 using the P1+×P1.

In the PCG solver for Stokes flow, the primary computational cost and memory bottleneck stem from solving the velocity-only submatrix system (Eq. 11). Therefore, to assess the GPU acceleration of the entire PCG solver, we first evaluated the performance of the GPU-based solver specifically for this submatrix system. Computational efficiency between CPU and GPU environments was compared using inner GMRES with the BoomerAMG preconditioner. Accuracy and convergence were verified through a manufactured solution prescribed on Ω=[0,1]2, with the corresponding source term derived from the prescribed solution. 15∇⋅τ=fwhereτ=2ηε˙  and  ε˙=12(∇u+(∇u)T)16η=exp⁡(cx),c=ln⁡(106) u, η, and f denote the velocity vector, viscosity, and prescribed source term, respectively. Viscosity increases exponentially along the x direction (Eq. 16), yielding a viscosity contrast of 106. The exact solution uexact (Fig. S1 in the Supplement) was prescribed as 17uexact=sin⁡(πx)sin⁡(πy)cos⁡(πx)cos⁡(πy),0≤x≤1,0≤y≤1.

The term f is derived by substituting the manufactured solution and viscosity into Eq. (15). 18f=2πexp⁡(cx)sin⁡(πy)(ccos⁡(πx)-πsin⁡(πx))-2π2exp⁡(cx)cos⁡(πx)cos⁡(πy)

Dirichlet boundary conditions were imposed on ∂Ω as u=uexact.

Performance of the AMG-preconditioned GMRES solver was evaluated for single-GPU and multi-core CPU implementations. The velocity fields obtained in the GPU and CPU runs were identical, with the L2-error relative to the exact solution remaining on the order of 10-10. Execution time and the GMRES iteration number were monitored. Comparison across various coarsening strategies showed architecture-dependent optimal choices (GPU versus CPU). In the GPU environment, HMIS converged in 17 iterations within 0.413 s, whereas PMIS required 20 iterations within 0.452 s (Table 1). In the CPU environment, PMIS, HMIS, and Falgout required 60 (21.92 s), 25 (9.76 s), and 13 (5.33 s) GMRES iterations, respectively, to reach convergence. Performance comparisons used the fastest setup on each architecture, with HMIS on GPU and Falgout on CPU. The GPU HMIS setup converged in four more GMRES iterations than the CPU Falgout setup, while achieving a 13× speedup in accumulated execution time (Table 2). In execution time comparisons between single-GPU and multi-core-CPU runs, the single-GPU HMIS setup ran faster than all CPU Falgout setups, except for the lowest resolution case (33 282 DOFs) (Fig. 1). For the largest DOF case (8 396 802 DOFs), the single-GPU required 0.413 s, corresponding to an approximately 13× speedup over the 32-core CPU time of 5.331 s.

We used the 2D SolCx benchmark, which has been widely employed to assess the stability and convergence of GPU Stokes solvers with strong viscosity discontinuities (Zhong, 1996; Duretz et al., 2011; Kronbichler et al., 2012; Gerya et al., 2013). The computational domain is the unit square Ω=[0,1]×[0,1] with free-slip boundary conditions on all four sides. The density field was prescribed to vary smoothly. In contrast, the viscosity exhibited a sharp jump at x=0.5 (Eq. 19). We set Ra to 0 and Rb to a spatially varying field to generate buoyancy (Eq. 20). A single composition field was adopted with C0=1. 19η(x)=1,0≤x≤0.5,106,0.5<x≤1.20Rb(x,y)=-sin⁡(πy)cos⁡(πx),0≤x≤1,0≤y≤1

The viscosity jump of 106 restricts the flow to the low-viscosity half, with the pressure discontinuous across the interface (Fig. S2). We compared the performance of the GPU-accelerated finite-element model with that of the 32-core CPU. The GMRES solver employed a default relative tolerance of 10-10. Performance was tested with varying GMRES relative tolerance and mesh resolution, based on the execution time of the PCG solver and velocity error relative to the analytical solution. The number of PCG iterations was fixed at five. After five PCG iterations, the L2-error relative to the analytical solution was smaller than 10-9 for both GPU and 32-core CPU configurations.

We found that the performance of the velocity-only subproblem translated directly into the PCG solver performance (Fig. 2a). HMIS on the GPU and Falgout on the CPU were the most effective choices. In all cases, PCG solver execution time on the single-GPU was shorter than that on the 32-core CPU (Fig. 2a). The GPU achieved an approximately 8× speedup over the 32-core CPU, demonstrating the high efficiency of the GPU architecture. Based on the comparison in Fig. 2a, the HMIS and Falgout coarseners delivered the best performance on a single-GPU and a 32-core CPU, respectively. Thus, the HMIS and Falgout coarseners were used for single-GPU and 32-core CPU performance comparisons in Fig. 2b and c. Then, we compared accumulated execution times for five PCG iterations on a single-GPU and 32-core CPU (Fig. 2b). The single-GPU recorded the shortest execution time at 2.25 s.

The performance evaluation with increasing problem sizes exhibited a similar trend (Fig. 2c). As the resolution increased, the gap between the GPU and CPU execution times widened. For all resolutions, the single-GPU delivered shorter execution times than the 32-core CPU environment. This advantage of the GPU was more pronounced for larger problems (Fig. 2c). For the highest resolution (1024×1024, 9 447 427 DOFs), the GPU required 2.25 s, while the 32-core CPU required 17.82 s, corresponding to a speedup of approximately 7.9×. The single GPU reached a throughput of 4.2×106 DOFss-1, compared with 5.3×105 DOFss-1 on the 32-core CPU. In the memory bandwidth benchmark (Table S1), the GPU sustains 824.0 GBs-1 and the 32-core CPU sustains 113.2 GBs-1, a ratio of 7.3×. The sustained ratio exceeds the theoretical peak ratio of 6.25× (960 and 153.6 GBs-1). The speedups measured across the Stokes benchmarks span 6–11× and cluster around the sustained ratio. The cost of PCG and GMRES is dominated by sparse matrix-vector products, data-parallel operations that map onto the GPU, where high memory bandwidth sustains the streaming access pattern. The unequal fractions of peak bandwidth attained by the GPU and CPU (86 % and 74 %, respectively) widen the sustained ratio beyond the peak ratio. The CPU and GPU versions of BoomerAMG further adopt different coarsening strategies that shift the number of iterations. The measured speedup depends on the cost per iteration and on the number of iterations. The sustained ratio serves as an approximate reference rather than an upper bound.

Figure 3a presents the accumulated execution times for the PCG solver on a single-GPU (red line) and a 32-core CPU environment (orange line). Both environments fixed the number of PCG iterations at three. Underworld2 (Mansour et al., 2020) was evaluated on a 32-core CPU using the default settings, including Q1/dQ0 quadrilateral elements, at a 2048×2048 resolution. Three pressure-correction iterations were used to solve the Schur-complement system. The velocity-only subsystem was solved using GMG-preconditioned GMRES. In Underworld2, the cumulative execution time over the three iterations was 7.7 s. For comparison with Underworld2, our FEniCS calculations used triangular P1/P0 elements at the same 2048×2048 resolution. Over three PCG iterations, the single GPU required 1.2 s, whereas the 32-core CPU required 9.96 s. The single GPU was therefore 6.42× faster than Underworld2 and 8.3× faster than the 32-core CPU implementation.

We measured the L2-error against the analytical solution with respect to PCG iteration number (Fig. 3b). All three configurations reduced the L2-error below 10-8 after the second pressure-correction iteration. The final converged error from the PCG solver was lower than that of Underworld2. This difference is attributed to the larger number of DOFs associated with the P0 element used in FEniCS (8 388 608 DOFs) compared to the dQ0 element used in Underworld2 (4 194 304 DOFs).

We measured execution times for individual stages of the PCG solver algorithm (solving velocity-only subsystem, vector update, and projection) to quantify differences between the 32-core CPU run and the GPU run (Fig. 3c). The single-GPU achieved an overall speedup of 8.3× relative to the 32-core CPU. The velocity-only subsystem stage dominated the total cost, with 7.53 s of the 9.96 s on the CPU and 1.04 s of the 1.20 s on the GPU, a speedup of 7.2×. The speed of the vector update and projection stages is limited by memory bandwidth. The high memory bandwidth of the GPU reduced the time of the two stages from approximately 2.4 s on the CPU to 0.16 s.

We measured the execution time to convergence while decreasing the GMRES relative tolerance from 10-4 to 10-10 (Fig. 4a). The single-GPU and 32-core CPU configurations attain the minimum execution times at 10-5, with execution time increasing at smaller tolerances. The single-GPU maintains the lowest execution time across the tested range, requiring 0.6–1.7 s compared with 5.4–15.2 s on the 32-core CPU. The number of outer PCG iterations decreases from seven at 10-4 to two at 10-5 and remains at two for smaller tolerances. The 7 PCG iterations at 10-4 cost more execution time than the 2 iterations at 10-5 on both configurations. A GMRES relative tolerance smaller than 10-4 delivers both the lower outer PCG iteration number and the shorter execution time. The single-GPU and 32-core CPU require an identical number of outer PCG iterations at each tolerance. For GMRES relative tolerances of 10-5 and smaller, the outer PCG iteration count remains fixed at two. Changes in total execution time therefore reflect the computational cost per outer iteration. We also calculated the L2 errors relative to the analytical solution over the tested tolerance range (Fig. 4b). The velocity error generally decreases as the GMRES relative tolerance is tightened, reaching the order of 10-11 at 10-8 and remaining at a similar level for tighter tolerances. Below 10-8, the execution time continues to increase without further improvement in velocity accuracy. The pressure error plateaus near 10-3 for GMRES relative tolerances below 10-8. A GMRES relative tolerance of 10-8 therefore provides converged accuracy without the additional computational cost associated with smaller tolerances. The outer PCG solver converged in two iterations across resolutions from 64² to 1024² on the single-GPU (Table 4) and 32-core CPU (Table 3), indicating mesh-independent convergence of the Schur-complement iteration. The lower convergence rate of the pressure L2 error with mesh refinement reflects the pressure discontinuity in the SolCx benchmark and the use of a continuous P1 pressure approximation.

We extended the classical 2D thermo-mechanical convection benchmark of Blankenbach et al. (1989) to a 3D non-rotating Boussinesq fluid to evaluate the performance and memory usage of the GPU solver in three dimensions. The thermal and compositional Rayleigh numbers were set to Ra=104 and Rb=0 in Eq. (1), isolating thermal effects by neglecting compositional buoyancy. A 3D rectangular domain (1×0.4×1) was employed to quantitatively compare computational performance and velocity-field accuracy between the GPU and CPU environments. The grid resolution was set to 100×40×100 using the MINI element (P1+×P1), yielding approximately 8.9 million DOFs for velocity and pressure. Free-slip boundary conditions were applied to all boundaries. For the temperature boundary conditions, nondimensional temperatures of T=0 and T=1 were prescribed at the top and bottom boundaries, respectively. The initial temperature field T0 consisted of a linear conductive profile with a superimposed long-wavelength perturbation of amplitude α=0.02 to initiate natural convection (Eq. 21). As time proceeds, thermal plumes rise from the lower boundary, which leads to a progressive reorganization of the internal temperature field and the heat-flow pattern at the boundaries (Fig. 5). 21T0=1-z+αcos⁡(πx)sin⁡(πz)

We utilised a fixed time step (Δt=0.00025) with the Crank–Nicolson scheme for time integration. We set the GMRES relative tolerance to 10-10 for the velocity-only subsystem on both GPU and CPU, continuing the PCG iterations until the divergence error (∥∇⋅u∥/∥u∥) fell below 10-7. The energy equation (Eq. 4) was solved using GMRES with a relative tolerance of 10-8, utilising BoomerAMG as the preconditioner, configured with HMIS for the GPU and Falgout for the CPU, respectively.

For the Stokes equations, the single-GPU (HMIS) achieved the lowest execution time with an average of 1.9 s (Fig. 6a; solid red line), compared with 17.1 s for the 32-core CPU (Falgout) (solid blue line). The speedup relative to the 32-core CPU was approximately 9×. At 8.87×106 DOFs, the average execution time per step corresponds to a throughput of 4.7×106 DOFss-1 on the single GPU and 5.2×105 DOFss-1 on the 32-core CPU. The average number of GMRES iterations per time step was 52 on the GPU and 23 on the 32-core CPU (Fig. 6a, coloured bars). The GPU exhibited shorter execution time per time step despite the iteration number being 2.3× higher than the CPU baseline. Memory usage measurements for the GPU environment included both VRAM and system memory (RAM) of the workstation (Fig. 6b). The average VRAM and system memory usage per time step were 21.2 GB (dotted red line) and 17.4 GB (solid red line), respectively, for a total of 38.6 GB. In the CPU environment, the average memory usage was 46.3 GB on the 32-core CPU, approximately 1.20× the total memory usage of the single-GPU configuration.

The temporal evolution of the global root-mean-square velocity (Vrms) exhibited complete consistency between the GPU and 32-core CPU environments (see Fig. S3). The measurements converged to 42.848, consistent with the value of 42.865 reported by Blankenbach et al. (1989). A Nusselt number (Nu) of 4.881 was calculated at the final time step in both configurations, matching the value of 4.884 presented in benchmark studies (Blankenbach et al., 1989; Tackley and King, 2003; King, 2009). To assess memory constraints associated with the multi-level generation of the AMG preconditioner, we measured the maximum memory usage for each computing environment with increasing mesh resolution (Fig. 6c). Measurements for the GPU environment included both VRAM and system memory. At the lowest DOFs (3 352 864), VRAM and system memory usage were 14.5 and 5.7 GB, respectively. At the highest DOFs (8 872 964), VRAM usage reached 45.5 GB, while system memory usage reached 20.6 GB. The 32-core CPU registered 15.3 GB at the lowest DOFs and 47.3 GB at the highest, exhibiting levels comparable to the GPU VRAM usage.

The Rayleigh–Taylor instability is a buoyancy-driven flow widely used as a benchmark for evaluating the performance of various advection schemes. Representative approaches include Particle-In-Cell (PIC) methods (e.g., Gerya and Yuen, 2003; van Keken et al., 1997) and level-set methods (e.g., Bourgouin et al., 2006). We set Ra=0 and Rb=1, indicating that only compositional buoyancy is activated. The initial interface between the two compositions is defined by a cosine function with an amplitude of 0.02 (Eq. 22). The level-set function was initialized (Eq. 23) from the interface at t=0. The composition field was set to 1 and 0 where the level-set function was positive and negative (Eq. 24), respectively. The level-set function was reinitialised every 500 steps. 22h(x)=0.02cos⁡(πx/0.9142)+0.223ϕ(x,y,t=0)=y-h(x)24C(x,y,t=0)=0,ϕ(x,y,t=0)≤0,1,ϕ(x,y,t=0)>0

For resolving complex velocity and composition structures, we adopted uniform grids with 512×512 resolution. The Taylor-Hood element (P2×P1) was applied to solve the Stokes equations. We compared the Vrms values computed by our FEniCS code with those from previous benchmark models (van Keken et al., 1997; Hillebrand et al., 2014). The advection equation for the level-set function was solved using GMRES preconditioned by HYPRE's BoomerAMG. The average per-step execution time was compared across GPU and CPU environments in the 2D benchmark (Fig. 7a). The GPU completed one step in 0.6 s on average (0.53 s for the Stokes equations and 0.07 s for advection), yielding speedups of ∼11× relative to the 32-core CPU, which required 6.3 s per step. The per-step cost was dominated by solving the Stokes equations (6.2 s for the 32-core CPU). The Stokes solve at 512×512 (2.36×106 DOFs) reached a throughput of 4.5×106 DOFss-1 on the single GPU and 3.8×105 DOFss-1 on the 32-core CPU. Despite the substantial improvement in computational efficiency enabled by GPU acceleration, the solution accuracy remains comparable to that of the CPU implementation. This agreement is demonstrated by the time evolution of Vrms, which closely matches the CPU simulation (Fig. 7b). Snapshot comparisons at key times (t=208, 699, 864, and 1500), when Vrms attains local maxima or minima, further support the model accuracy. In addition, the GPU-based model reproduces the reference Vrms values from Hillebrand et al. (2014) and van Keken et al. (1997). We also tested the Rayleigh–Taylor instability in 3D on a 65×65×65 mesh with the MINI element. The single GPU delivered a speedup of ∼9× relative to the 32-core CPU across the entire time range, while the two configurations produced identical Vrms values (Fig. S4).

In geodynamic numerical modelling, the sticky-air technique is a widely used approach for approximating a free surface. Crameri et al. (2012) suggested that to correctly represent a free-surface boundary condition, the sticky-air layer requires a viscosity contrast of 10-5 relative to the underlying material and a thickness of 100 km, or alternatively a viscosity contrast of 10-4 and a thickness of 200 km. In our study, we set the viscosities of the lithospheric, mantle, and sticky-air layers (with a thickness of 100 km) to ηlitho=1023 Pas, ηmantle=1022 Pas, and ηair=1018 Pas, respectively. Buoyancy in the sticky-air benchmark is purely compositional. The isothermal configuration of Crameri et al. (2012) fixes Ra=0 in the absence of a temperature field, leaving Eq. (4) outside the solution. The benchmark constrains the free-surface response alone, with the thermo-mechanical benchmark held separate. The dimensional parameters for the sticky-air benchmark are presented in Table S3. We discretized the Stokes system using P2×P1 finite elements on a 1400×800 mesh. The boundary between the lithosphere and the sticky-air was assigned a cosine-shaped perturbation. We compared the maximum topography with the analytical solution from Crameri et al. (2012). The initial composition field (C) is defined as three layers separated by the upper and lower interfaces of the lithosphere (Eq. 25). C1, C2, and C3 denote sticky-air, lithosphere, and mantle, respectively. Two level-set functions (ϕ1 and ϕ2) were defined for the two interfaces (Eqs. 26 and 27). 25C=C1,ϕ1(x,y,t=0)≤0C3,ϕ2(x,y,t=0)<0C2,else26ϕ1(x,y,t=0)=700+7cos⁡2πx2800-y27ϕ2(x,y,t=0)=y-600

An initial perturbation imposed on the lithosphere generates a lateral density gradient, which drives upwelling in the centre of the domain and downwelling on both sides, forming two convection cells (Fig. 8a). Advective transport progressively flattens the initial topographic perturbation. At 5.77 kyr, the decaying perturbation weakens the flow in the two convection cells (Fig. 8b). The surface recovers an almost flat geometry by the final time of 100 kyr (Fig. 8c). In contrast, the divergence-free dynamics of mantle and lithosphere flow advect the initially horizontal lithosphere–mantle boundary, producing a secondary perturbation by the final time (Fig. 8c). The maximum topography over the simulation time was identical between the GPU and CPU executions (Fig. 8). The topographies from the CPU and GPU are consistent with the analytical solution and the reported curve of Hillebrand et al. (2014), indicating that the physical solution on GPU remains consistent despite the larger number of GMRES iterations.

Execution time comparison over the total steps shows that the GPU consistently outperformed the 32-core CPU (Fig. 9a). The GPU (HMIS) execution time remained stable within 4.7–11.1 s, whereas the CPU (Falgout) execution time increased from 30 s to more than 100 s (Fig. 9a; solid blue line), corresponding to an overall speedup of roughly 7–8× for the GPU (solid red line). At 1.01×107 DOFs, the per-step execution time corresponds to a throughput of 0.9–2.1×106 DOFss-1 on the single GPU and 1.0–3.4×105 DOFss-1 on the 32-core CPU. The number of PCG iterations varied with simulation time, while the GPU and CPU generally required similar numbers of PCG iterations at each time step (Fig. 9b). Time steps with a larger number of PCG iterations also exhibited a larger number of GMRES iterations within the iteration loop. The GPU and CPU required 250–650 and 150–410 GMRES iterations, respectively.

The subduction application targets 2D nonlinear viscous Stokes flow incorporating a plate interface where one plate descends into the mantle. The computational domain is a rectangle with a width of 3960 km and a depth of 660 km, which is then nondimensionalized (Enns et al., 2005; Capitanio et al., 2010; Sharples et al., 2014). Characteristic scales for nondimensionalization are listed in Table S4. The Stokes system was discretized with P2×P1 elements on a 1200×200 structured grid, where each rectangular cell was subdivided into two triangular elements along the lower left to upper right diagonal. The compositional structure is partitioned into five regions: mantle, overriding plate, upper slab layer, lower slab layer, and slab core. Four level-set functions, ϕ1 through ϕ4, delineate the region interfaces.

The upper mantle serves as the reference fluid with viscosity ηUM=1 and ρ=0. The overriding plate has the viscosity ηOP=400, yielding a viscosity contrast of 400 relative to the upper mantle, while its density matches the mantle value. The viscosity of the upper slab layer (ηUS) follows a Mohr–Coulomb plasticity law (Eq. 28). 28ηUS=12τyε˙II,ε˙II=12ε˙:ε˙ τy, ε˙, and ε˙II denote the plastic yield stress, strain-rate tensor, and the square root of its second invariant, respectively. τy was set to 0.06. ηUS is bounded between 1 and 1000 to improve numerical stability. Additionally, at nondimensional depth y<0.7, weakening is imposed by prescribing η=50. The viscosities of the slab core (ηSC) and lower slab (ηLS) are 1000 and 50, respectively. Mohr–Coulomb plasticity in the upper slab layer introduces nonlinear viscosity into the Stokes system, requiring Picard fixed-point iteration with a relative tolerance of 10-3. As the viscosity contrast increases and compositional interfaces become more complex, the condition number of the discretized Stokes system tends to increase, degrading Krylov solver convergence. We therefore approximated the condition number at each time step using the ratio of the largest to smallest eigenvalues of the stiffness matrix and examined its relationship with the GMRES iteration number and execution time. An initial slab perturbation, pre-subducted to 200 km depth with a 29° dip, triggers subduction (Fig. 10a). The slab maintains a steep dip and ultimately reaches the base of the upper mantle (Fig. 10b).

The performance gap between the GPU and CPU persists in the subduction model. Over the full time-stepping sequence, the GPU reduces execution time by a factor of 6–7 relative to the 32-core CPU run (Fig. 11a). In geodynamic simulations, time evolution increases the spatial complexity of density and viscosity fields, often accompanied by large material contrasts. Such conditions worsen the system conditioning, increasing the condition number (purple line in Fig. 11a). Higher condition numbers require more GMRES iterations. In the subduction model, the estimated condition number increases with time step (Fig. 11a). As the condition number increases, execution time increases in both the GPU and CPU runs (Fig. 11b). However, the regression between GMRES iterations per step and condition number shows a steeper slope for the GPU (Fig. 11c). Despite this steeper slope, the GPU remains approximately 6× faster in execution time, as its lower per-iteration cost keeps the total execution time shorter. At 2.17×106 DOFs, the throughput ranged from 1.1×105 to 1.8×105 DOFss-1 on the single GPU and from 1.4×104 to 3.1×104 DOFss-1 on the 32-core CPU. The Picard iterations within each step lower the throughput by an order of magnitude relative to the linear benchmarks.

A multi-GPU configuration is essential to overcome the VRAM limit of a single-GPU. At high resolution, the growth in DOFs substantially increases the memory requirements for the stiffness matrix, the preconditioner, and the global force vector. Multi-GPU execution distributes VRAM usage via MPI-based domain decomposition, which also enables additional speedup through parallel execution. Inter-GPU communication and synchronization, however, introduce extra computational costs. We evaluated how throughput varies with resolution by comparing single-GPU and two-GPU configurations across six benchmarks in Sects. 4.1–4.6. In the velocity subproblem using P2 elements, the single-GPU delivered higher throughput than the two-GPU configuration at the lowest resolution of 128×128 (1.32×105 DOFs) (Fig. 12a). In the low-resolution regime, communication and synchronization overhead exceeded the performance gain from parallelization. At resolutions of 256×256 and higher, however, the two-GPU configuration delivered higher throughput than the single-GPU. In the high-resolution regime, the increased computational workload sufficiently outweighed the overhead, yielding a clear performance advantage for two-GPU runs. In addition, memory capacity limited the single-GPU to a maximum resolution of 1152×1152 (1.06×107 DOFs), whereas the two-GPU configuration ran up to 1536×1536 (1.89×107 DOFs). We compared the throughput of single-GPU runs with two-GPU runs across mesh resolutions for the SolCx benchmark using P2×P1 elements (Fig. 12b). In the high-resolution regime (>512×512, corresponding to 2.36×106 DOFs), the two-GPU configuration outperformed the single-GPU by a factor of 1.03 (for 640×640) to 1.31 (for 1024×1024). The two-GPU configuration supported a maximum resolution of 1664×1664 (24 936 707 DOFs), whereas the single GPU was limited by memory capacity to 1152×1152 (11 955 459 DOFs).

For the 3D thermo-mechanical convection using the MINI element, the grid resolutions in the x and z directions were held fixed at 100 cells (Fig. 12c), while the y direction resolution increased from 10 to 80. The two-GPU configuration delivered higher throughput at every resolution. The throughput ratio increased from 1.07 to 1.64 as the resolution increased. The 2D Rayleigh–Taylor instability test with P2×P1 elements followed the same pattern as the 2D SolCx benchmark (Fig. 12d). The single-GPU delivered higher throughput at low resolution, whereas the two-GPU throughput exceeded the single-GPU throughput beyond 512×512. In the sticky-air benchmark, the two-GPU configuration delivered higher throughput than the single-GPU at all resolutions (Fig. 12e). The throughput ratio ranged from 1.5 to 2.1. In the subduction model with nonlinear viscosity, the two-GPU throughput exceeded the single-GPU throughput beyond 900×150. Then, the two-GPU throughput reached 1.4 times the single-GPU throughput (Fig. 12f). The single-GPU delivered higher throughput at low resolution in the SolCx, 2D Rayleigh–Taylor, and subduction benchmarks. Once the DOFs exceeded a threshold, the computational workload dominated, reducing the relative impact of overhead, and the two-GPU configuration became more effective. In 3D problems, the number of DOFs increases more rapidly with mesh resolution, yielding a consistent advantage for the two-GPU configuration across the tested resolutions.

In this study, we present a practical strategy to reduce bottlenecks in a single-GPU workflow, where one CPU core assembles matrices and the GPU performs iterative solves. Before performing GPU-based PCG solves, the L2-projection mass matrix, gradient operator, and divergence operator are assembled once on the CPU and stored in GPU VRAM. Pre-assembly of the three operators removes the assembly cost from every PCG iteration. The end-to-end throughput of each time step therefore increases. The approach becomes increasingly beneficial when many PCG steps are nested inside Picard iterations. The benefits are most pronounced for iteration-heavy workloads, such as the Picard iterations for the nonlinear-viscosity subduction model (Sect. 4.6). We note that the stiffness matrix must still be reassembled when viscosity changes between time steps. Storing preassembled operators increases VRAM requirements as the number of DOFs increases. For an incompressible Stokes system on a fixed mesh, the gradient and divergence operators for velocity–pressure coupling remain constant across iterations. For example, at a 100×40×100 resolution with the MINI element for the 3D thermo-mechanical convection benchmark (Sect. 4.3), storing these operators and associated vectors requires approximately 1.3 GB of VRAM (Table S2), which is a modest fraction of the ∼46 GB peak VRAM usage recorded.

Nonlinear Stokes systems with strain-rate-dependent viscosity require repeated assembly of the stiffness matrix. FEniCS 2019 performs each assembly on the CPU, adding a CPU–GPU data transfer at every nonlinear iteration. In multi-GPU runs, stiffness matrix assembly can be distributed across multiple CPU cores via MPI parallelism, offering additional assembly-speed improvements. GPUs are designed to maximise performance by executing large numbers of operations simultaneously. When many CPU cores assemble MPI-distributed matrix data and offload computations to a small number of GPUs (two in our study), technologies such as NVIDIA MPS (Multi-Process Service) become essential. MPS collects computation requests from multiple MPI processes and forwards them to the GPU. The GPU then executes the distributed matrix operations concurrently via MPS. In both 2D and 3D tests, the 16-core CPU with one or two GPUs achieved higher end-to-end throughput than the 1-core CPU with a single GPU. For the sticky-air benchmark at a 1400×800 resolution, the end-to-end throughput of the 16-core CPU with two GPUs was twice that of the single GPU with a 1-core CPU (Fig. S5). Across the time-dependent benchmarks, the single GPU with a 1-core CPU delivered 1.14–3.46× the end-to-end throughput of the 32-core CPU. The 16-core CPU with two GPUs delivered 5.19× the end-to-end throughput of the 32-core CPU (Table S5). This indicates that for 3D high-resolution problems, an MPS-based multi-core CPU and multi-GPU setup is most effective. A key limitation of the multi-GPU implementation based on FEniCS is the incompatibility between FEniCS's high-level function layer and GPU-aware MPI protocols. When GPU-direct communication is enabled, ghost-node values at MPI partition interfaces are incorrectly set to zero. FEniCS's topology handling and DOF mapping depend on system-memory data structures, while the underlying PETSc backend operates on CUDA vectors. During ghost-value synchronization, FEniCS relies on system memory for scatter/gather operations, whereas GPU-aware MPI relies on device-pointer communication (Mills et al., 2021). This mismatch causes memory incoherence, preventing correct updates of interface values. We addressed this issue by using native PETSc calls for pre-assembly. We constructed the force vector directly via differential and projection operators, bypassing the problematic FEniCS layer. This approach preserves correct boundary synchronization via PETSc-GPU communication protocols while maintaining GPU-aware MPI performance.

Recently, finite element codes have incorporated GPU acceleration through global sparse matrix assembly and matrix-free evaluation. GPU-accelerated finite-element assembly has been demonstrated in FEniCS (Trotter et al., 2023). Matrix-free GPU evaluation is available in deal.II and MFEM (Kronbichler and Ljungkvist, 2019; Anderson et al., 2021). MFEM additionally assembles global sparse matrices on the GPU. A longer-term goal is a fully GPU-resident workflow that performs both assembly and linear solves on the GPU, further reducing transfer overhead. Until GPU-side global matrix assembly becomes widely adopted in the geodynamics community, our approach offers a practical alternative by keeping assembly and host–device transfer costs modest while delivering ∼5× the end-to-end throughput of the 32-core CPU-only configuration with a 16-core CPU and two GPUs.

Multi-GPU execution is essential for high-resolution geodynamics, as the approach shortens execution time while extending the maximum mesh resolution and total DOFs depending on VRAM capacity. Our benchmark tests show a clear path towards higher-resolution Stokes modelling in both 2D and 3D through single-GPU speedups and multi-GPU scaling. Recent 2D geodynamics studies increasingly use finer meshes to represent subduction zones, plate boundaries, shear zones, and boundary layers near free surfaces without sacrificing local resolution. For example, a high-resolution rifting setup uses 0.5 km spacing at depths shallower than 200 km (Yu et al., 2025). High-resolution 2D subduction and continental collision studies employed a mesh of 1024×512 elements, extending to 2048×1024 elements for resolution-sensitivity tests (Laik et al., 2023). Under such high-resolution settings, Stokes DOFs and preconditioner memory requirements increase sharply, making GPU VRAM capacity and solver execution time key constraints. In our 2D SolCx benchmarks, the two-GPU configuration outperformed a single-GPU configuration above 512×512 elements. Between 640×640 and 1024×1024, the two-GPU configuration delivered 1.03–1.31× the single-GPU throughput. These speedup factors enable higher resolution 2D rifting and subduction simulations under tight VRAM limits.

In 3D, the number of DOFs increases more rapidly with mesh resolution, leading to memory bottlenecks due to the stiffness matrix, preconditioner, and force vector. Building the preconditioner also consumes extra VRAM through coarsening levels, pushing a single-GPU to its memory limit at relatively modest resolutions. Ultra-large 3D Stokes problems therefore require massive multi-GPU execution. Räss et al. (2022) used a pseudo-transient method on 2197 GPUs to solve 3D variable-viscosity Stokes flow at 49953 resolution, reaching 1.2 tera-DOFs. The pseudo-transient framework targets ultra-high-resolution finite-difference simulations at extreme multi-GPU scale. We propose that multi-GPU scaling directly supports larger DOF targets within available VRAM. An extrapolation based on our 3D DOF scaling suggests that eight RTX 6000 Ada cards (48 GB VRAM each) would support roughly 7.0×107 DOFs. Replacing this configuration with RTX PRO 6000 cards (96 GB VRAM each) would yield a total of 768 GB VRAM, enabling over 1.4×108 DOFs under the same extrapolation.

Krylov solvers for the Stokes system benefit mainly from high memory bandwidth and large device memory. HPC-class GPUs such as the NVIDIA H200 provide 4.8 TBs-1 and 141 GB for the two quantities, approximately 5× and 2.9× the values of the RTX 6000 Ada, respectively. The higher bandwidth of the H200 is expected to lower the cost of operations limited by memory bandwidth. The overall gain for the Stokes solver requires direct measurement on the H200. The larger device memory increases the maximum problem size per GPU. H200-class GPUs offer a path toward higher-resolution simulations of global mantle convection and complex crustal deformation.

GPU acceleration reduces the execution time of Stokes solves in both single- and multi-GPU configurations. Single-GPU execution reduces the time per solve, while multi-GPU execution also increases the maximum DOFs that can be handled by distributing memory demand across GPUs. This combination enables high-resolution forward modelling and inverse problems that are difficult to run within the execution time and memory limits of conventional CPU clusters. Multi-GPU configurations further support scaling from local domains to global models by combining larger total VRAM with parallel execution.

Accurate simulation of narrow shear zones or thin lower-mantle thermal boundary layers in a 3D spherical-shell domain often requires grid spacing on the order of tens of kilometers (Taiwo et al., 2023; Murphy and King, 2024), which drives the Stokes system to very large DOF. In this setting, the GPU-based PCG solver tested in our study remains effective at large 3D scales and is expected to be important for future global simulations. Li et al. (2023) reconstructed the subduction history of the North American plate using plate-reconstruction-driven data assimilation and emphasised the high computational cost associated with realistic 3D buoyancy and viscosity structures. The study combined seafloor-age constraints and lithosphere-density profiles to build spatially variable buoyancy forcing and viscosity fields. The resulting configuration increased computational cost by more than an order of magnitude relative to simplified models. Such cost limits further extensions, including coupling with surface processes or increasing spatial resolution in global simulations. Faster Stokes solves on GPUs, together with multi-GPU scaling that supports larger DOF cases, are therefore critical for making such high-fidelity data-assimilation models more practical.

GPU acceleration also reduces the cost of adjoint inversion. Adjoint inversion minimises the misfit between an observed mantle state and a model prediction by iteratively updating initial conditions or model parameters. Each iteration computes the misfit through a forward simulation, then solves the adjoint equations backwards in time to obtain sensitivity information used to update the model. Since one iteration requires both forward and adjoint integrations over the full time window, the total cost scales with the number of iterations and the length of the assimilation period. Colli et al. (2020) limited the inversion window to 50 Myr (rather than 100 Myr) to manage the computational cost and excluded an inner loop for calibrating parameters (e.g., viscosity). The GPU speedups demonstrated in our study increase the number of forward-adjoint cycles that fit within a fixed runtime budget. This improvement supports longer assimilation windows. The same improvement also makes repeated parameter-calibration loops, including viscosity updates, more practical. GPU acceleration therefore provides the computational basis for extended inversion workflows that better constrain uncertain material properties and initial conditions in high-resolution geodynamics.

In computational geodynamics, one of the state-of-the-art Stokes solvers is based on Schur-complement block preconditioners, which are widely used in major community codes such as ASPECT (Kronbichler et al., 2012). A block preconditioner handles the saddle-point problem, with an AMG (or GMG) preconditioner applied to the velocity block and the pressure Schur complement approximated by a viscosity-weighted mass matrix or a BFBT-type scheme. This configuration delivers mesh-independent convergence and excellent scalability on large parallel systems (Benzi et al., 2005; May and Moresi, 2008; Rudi et al., 2017). In contrast, the PCG solver adopted in this study applies a conjugate-gradient iteration to the pressure Schur complement system. The outer iteration stores only a fixed number of vectors, requiring less memory than block-preconditioned FGMRES, which accumulates Krylov vectors as the iteration proceeds. However, the inner velocity subproblem is solved with GMRES and requires additional Krylov-vector storage. As a result, the overall memory saving is smaller than that suggested by the outer iteration alone.

To evaluate this solver's effectiveness, we compared a FEniCS-based PCG implementation with ASPECT's block preconditioned solver for the 3D thermo-mechanical convection benchmark (Table S5). The FEniCS implementation was tested on a two-GPU configuration with a 16-core CPU, a single-GPU configuration with a 16-core CPU, and a 32-core CPU, while ASPECT was run on the 32-core CPU. The ASPECT calculation used the single-Advection, single-Stokes nonlinear solver scheme with the block-AMG Stokes solver. The Stokes linear solver tolerance was set to 10-2, with a GMRES restart length of 100, while the temperature solver tolerance was 10-6. To enable a fair comparison with ASPECT, which provides Q2/Q1 elements, we used a P2/P1 discretization in FEniCS with a resolution of 40×40×40. The simulation covered a total nondimensional time of 0.25 over 1000 steps. On the 32-core CPU, ASPECT required an execution time of 471 s to solve the Stokes system over the full elapsed run. Our PCG solver on the 32-core CPU required an execution time of 961 s, indicating that ASPECT's block-preconditioned Stokes solver outperforms our PCG solver on CPUs in this configuration. However, with a single GPU and a 16-core CPU, the same case required 247 s, corresponding to speedups of approximately 3.89× and 1.91× relative to our 32-core PCG run and the 32-core ASPECT run, respectively. With two GPUs and a 16-core CPU, the Stokes solving time decreased to approximately 126 s, delivering an additional 2.0× speedup over the single-GPU case and overall speedups of 7.63× and 3.74× relative to our 32-core CPU run and ASPECT, respectively.

A block-preconditioned formulation solves the full Stokes system with block preconditioned FGMRES, and a carefully tuned GPU implementation of this strategy may ultimately be faster than the PCG splitting approach. At the same time, the full Stokes system is larger and each FGMRES iteration must store multiple Krylov vectors, which can lead to higher memory usage, especially in multi-GPU configurations. The velocity-only submatrix is symmetric positive definite, allowing CG to replace GMRES for the velocity subproblem. CG stores fewer Krylov vectors than GMRES, whereas the dominance of the BoomerAMG hierarchy in solver memory limits the overall memory reduction. Selecting the most effective solver requires a systematic comparison of PCG splitting and block-preconditioned FGMRES in execution time and memory usage.