In unstructured meshes, element connectivity must be explicitly stored. Each tetrahedral element in Fluidity-Atmosphere records four vertex indices. Physical variables are stored in a unified Fields data structure organized as (number of vertices × dimension) arrays: scalars (1D), vectors (3D), and tensors (9D). The same layout is adopted in GPU memory to ensure contiguous access and consistent indexing (Fig. 2). Common fields such as density and temperature are stored as scalars, positions and velocities as vectors, and viscosity terms as tensors.

The element-wise computations in the FEM are intrinsically independent, making them particularly suitable for massively parallel execution on GPU architectures. Unlike spectral methods that involve global communication or high-order finite-difference schemes relying on extended stencils, finite-element computations are confined strictly within each element boundary. It is important to clarify that while Fluidity-Atmosphere utilizes a mixed CG/DG discretization, the GPU acceleration in this study focuses exclusively on the Continuous Galerkin (CG) operators (e.g., for the momentum and advection-diffusion equations). In the CG formulation, the numerical integration to evaluate local matrices remains mathematically entirely local. Unlike DG methods, which require calculating inter-element fluxes and jumps over facets, the CG element-wise computations do not introduce cross-element mathematical dependencies during the integration phase. The global coupling arising from shared mesh nodes is manifested exclusively during the global matrix assembly phase through indirect memory access.

This strong locality offers two principal advantages: (1) all element-wise computations can proceed independently, eliminating inter-thread communication during the compute phase; and (2) the workload is naturally balanced across elements, minimizing synchronization overhead. Consequently, a “one-thread-per-element” parallelization strategy is adopted. Each GPU thread handles one element. In our implementation, thread block sizes of 128 and 256 were both utilized, with the specific parameter for each kernel determined empirically via NVIDIA Nsight Compute. For the target mesh comprising 554 394 elements, a block size of 128 yields 4332 blocks, while 256 yields 2166 blocks. Given that the NVIDIA A100 GPU features 108 Streaming Multiprocessors (SMs), each with a theoretical maximum of 2048 resident threads, both configurations generate sufficient numbers of thread blocks to provide approximately 2.5 execution waves across the GPU, which helps maintain high occupancy and effectively hide memory latency.

With the data layout on GPUs established, the next step is to implement the core finite element operation: coordinate transformation. Using a reference element (in this study, a tetrahedron), shape functions and Gauss points are consistently defined. In Fluidity-Atmosphere, each physical element employs 11 Gauss points: four vertices, six edge midpoints, and the centroid (Fig. 3). For each Gauss point, the transformation routine evaluates the integrand and multiplies it by the predefined weight.

Even though element-independent and theoretically parallelizable, this kernel is computationally intensive and one of the most frequently executed routines in the model. For every iteration and every element, it performs numerous small matrix operations: matrix multiplications, adjoint and determinant evaluations, and transformations of derivative matrices.

Algorithm 1 outlines the transform_to_physical procedure, where input and output arrays are stored contiguously to maximize memory efficiency. Crucially, the algorithm incorporates a conditional optimization for linear (P1) elements: because standard affine mappings yield a constant Jacobian matrix, expensive operations (Jacobian construction, determinant, and inverse) are evaluated only at the first Gauss point (gi=1) and cached.

The inner loop over Gauss points is deliberately retained to preserve framework generality for higher-order elements. Furthermore, rather than precomputing and storing these geometric quantities, which would exacerbate memory traffic in this heavily memory-bound framework, we employ a “compute-on-the-fly” strategy, trading abundant GPU arithmetic cycles to conserve precious memory bandwidth .

Element-wise integration routines constitute the computational core of unstructured finite-element models, as they are responsible for evaluating the contributions of each element to the global system. Profiling of Fluidity-Atmosphere indicates that these kernels are among the most frequently executed routines. During each nonlinear iteration, they are called once per element per physical equation, resulting in millions of invocations per timestep. As each call involves multi-dimensional tensor operations and numerical integration over multiple Gauss points, the accumulated cost dominates both the element-computation phase and the overall runtime of the model. Therefore, optimizing element integration is critical to achieving substantial end-to-end performance gains. The principal integration functions are as follows:

dshape_tensor_dshape couples shape function gradients with tensor fields, supporting stabilization terms and physical field coupling.

Among these functions, the dshape_tensor_dshape function is the most computationally intensive, as it involves both tensor-vector and vector-vector multiplications for every integration point. On the GPU, this procedure is parallelized at the thread level, with each thread processing one element. Beyond kernel-level parallelization, a series of GPU-specific performance tuning techniques are applied to completely exploit the hardware potential. These optimizations are guided by profiling by employing NVIDIA Nsight Compute, focusing on reducing memory latency, register pressure, and control overhead:

Memory optimization: The __restrict__ qualifier is applied to pointer variables to eliminate aliasing and enable more aggressive memory-access optimizations. Small __device__ functions are annotated with __forceinline__ to reduce function-call overhead.

These measures are reflected in Nsight Compute reports as improved register utilization and reduced memory bottlenecks. Loop unrolling and templating yield substantial enhancements in small-scale computations. In combination with GPU-parallel element integration and small-matrix solvers, these optimizations form a cohesive strategy that maximizes performance while maintaining full consistency with the original CPU implementation.

It should be noted that the CPU baseline used for comparison represents the official, unmodified production code of Fluidity-Atmosphere, compiled with aggressive optimization flags -O3 -ffast-math. Under these flags, modern CPU compilers automatically apply substantial loop unrolling and vectorization. However, due to the warp-based execution model and strict register allocation of GPUs, the CUDA compiler frequently requires explicit manual directives (e.g., #pragma unroll) and compile-time size resolution (via templates) to achieve optimal instruction scheduling and register reuse. Thus, these GPU-specific manual optimizations are implemented to fully unlock the hardware's architectural potential rather than to introduce an algorithmic disparity. The optimized code is listed in Appendix A.

In addition to numerical integration, the construction of stabilization terms involves numerous small-scale linear algebraic operations, such as solving 6 × 6 systems or computing eigen-decompositions of 3 × 3 matrices. Even though these operations are apparently lightweight, they are called repeatedly for every element and Gauss point, resulting in a substantial cumulative cost. On CPUs, relying on standard library routines such as LAPACK's DSPEV or Fortran's intrinsic solvers for these operations is highly inefficient. While a small 6 × 6 matrix easily fits into the L1 data cache, gathering these matrices from an unstructured mesh via indirect addressing disrupts spatial locality. Furthermore, these independent tiny matrices offer minimal opportunity for data reuse, limiting cache efficiency. From a software perspective, pre-compiled external library routines typically cannot be automatically inlined by the compiler within the tight element loop. Invoking them millions of times introduces profound function-call overhead, and their internal argument validation and branching dwarf the actual floating-point arithmetic.

To address this challenge, this study implements dedicated GPU-based small-matrix solvers and eigen-decomposition kernels as inline __device__ operators.

Small linear systems (e.g., 6 × 6): These are solved on GPUs using Gaussian elimination. The input is an augmented matrix A (in column-major layout, including the right-hand side) with dimensions m×n. Pivoting and row exchanges are applied to control numerical errors, whereas column-major storage and in-place operations reduce register pressure and facilitate the alignment with GPU memory characteristics.

These GPU-implemented solvers are lightweight and reusable across kernels. The modular __device__ design also promotes code reuse across physical modules and supports further fusion with integration kernels.

Following local integration, the algorithm accumulates (Ae, Fe) into the global matrix A and vector F according to the connectivity information of the mesh. This procedure, widely recognized as the global assembly, transforms element-level contributions into a consistent global representation based on the mapping between local and global degrees of freedom, as shown in Algorithm 2. Here, ε is the set of elements and e is an element. The elem function performs element-wise computations to get element matrices. The L function retrieves the node numbering.

In contrast to structured-grid models, where mesh nodes are arranged in a regular (i,j,k) indexing order and memory access can be performed sequentially and predictably, unstructured meshes lack geometric regularity. The connectivity of each element varies depending on its neighbours, and the mapping between local and global indices must be retrieved indirectly through lookup tables. Consequently, the assembly phase in unstructured FEM involves frequent non-contiguous global memory reads and writes, with irregular strides between consecutive memory locations. This destroys spatial locality and drastically increases cache miss rates, thereby amplifying the ratio of memory access time to floating-point computation.

On GPU architectures, these irregular access patterns exacerbate performance degradation. Concurrent threads typically need to update overlapping entries of the global sparse matrix, requiring synchronization or the use of atomic operations to ensure correctness. The combined effects of irregular global memory traffic, low cache reuse, and write conflicts make matrix assembly a highly memory-bound process.

Consequently, global matrix assembly presents two major challenges: (1) the imbalance between memory traffic and arithmetic intensity, caused by the dominance of irregular global data movement over computation; (2) the difficulty of ensuring data consistency during concurrent updates. Cumulatively, these factors make matrix assembly another critical performance bottleneck, and thus a central focus of GPU parallel optimization in this work.

Fluidity-Atmosphere employs different sparse storage formats for different types of physical fields. For scalar field matrices, the compressed sparse row (CSR) format is used. For vector field matrices such as three-component velocity, the block-CSR (BCSR) format is adopted.

For a scalar field, the local stiffness matrix can be expressed as Eq. (1): 1Kij=∫Ωe(∇Ni)TC(∇Nj)dΩ where Ni and Nj are basis functions, Ωe is the integration domain of the element, and C is the associated tensor. For scalar fields, Ni is a scalar and Kij is a scalar integral, resulting in a 4 × 4 local element matrix. By obtaining the global node indices ni and nj corresponding to the local nodes i and j, the result Kij is updated in the global matrix entryy(ni,nj).

The global matrix is typically stored in the CSR format, which consists of three arrays: values (non-zero entries), col_index (column indices), and row_pointers (row offsets) . Within a row range, the target column can be located using binary search.

For vector fields, each node contains three degrees of freedom (DOFs). Thus, while assembling the local stiffness matrix, all interactions among the nodes and their DOFs must be considered. This results in a set of 4 × 4 blocks, each of size 3 × 3: 2K=K11K12K13K14K21K22K23K24K31K32K33K34K41K42K43K44.

Each element of the element matrix Kij is actually a 3 × 3 matrix. 3Kij=abcdefghi.

Here, Fluidity adopts the BCSR format, which is structurally similar to CSR, but stores continuous block data in the value array. This block-based representation better supports sparse operations for vector fields.

In our GPU parallelization, the contributions of each mesh element are computed and stored in parallel, with each GPU thread responsible for one element. A dedicated kernel is launched to assemble element matrices.

In the implementation, each thread iterates over the entries Kij of the local element matrix and looks up the corresponding global non-zero indices. Given the global node indices ni and nj, the row_pointers array is first used to locate the row range [row_pointers(ni), row_pointers(ni+1)). Within this range, the column index nj is searched using binary search. To enhance the efficiency, the implemented GPU assembly function consistently adopts binary search.

A significant challenge arises during parallel write-back, where race conditions may occur if multiple threads attempt to update the same non-zero entry simultaneously. An atomic operation is indivisible and guarantees consistency by preventing interference from other threads. Modern GPUs provide hardware-level support for atomic addition, enabling conflict-free concurrent updates without explicit locks. In this work, the atomic add approach is adopted for its efficiency and simplicity. The CUDA parallel assembly code is listed in Appendix B.

To summarize the complete execution pipeline, the global matrix assembly occurs entirely within the GPU device memory. Following the element-wise computations, the local element matrices are accumulated into the global sparse matrix arrays (i.e., the values array in CSR/BCSR format), which are pre-allocated on the GPU. This is achieved using the aforementioned parallel assembly kernel equipped with hardware-level atomicAdd operations. Upon the completion of the assembly kernel, the fully constructed global matrix arrays are transferred back to the CPU host memory via PCIe. The assembled matrix is then handed over to the CPU-bound PETSc library to perform the subsequent large-scale linear system solve.

Similar to the global matrix, the RHS vector is assembled using an element-wise parallel strategy. Each GPU thread computes the nodal contributions of one element and accumulates them into the corresponding global vector entries.

The implementation first determines the indices of the nodes belonging to each element, then loads the nodal values, and finally applies atomicAdd operations to ensure correctness during concurrent accumulation. For cases where multiple variables are associated with a node, appropriate offsets are added to the corresponding positions. The Appendix C illustrates the procedure of RHS assembly, which achieves efficient large-scale parallel accumulation while preserving correctness.

This section presents a systematic analysis of the performance of the proposed GPU parallelization strategies. Experiments were conducted to first validate correctness on both CPU and GPU platforms, and then to evaluate performance in four aspects: element-wise computation, global matrix assembly, data transfer with computation overlap, and overall model performance. The GPU and CPU experimental environments are shown in Table 2. All CPU baseline evaluations were executed using the specified number of CPU cores (MPI processes) on a single physical AMD EPYC 7713 processor.

GPU acceleration was applied to the momentum equation, advection equation, and stabilization term construction of Fluidity-Atmosphere. Then, the GPU-accelerated version was validated for correctness and evaluated for performance to ensure both accuracy and efficiency.

For validation of correctness, the 3D idealized mountain wave test case was used. The mountain wave test serves as a crucial validation procedure for assessing the dynamic framework model of the Fluidity-Atmosphere model in simulating orography-induced airflow. By comparing numerical results against theoretical solutions of idealized orographically forced flows, this test effectively evaluates the capability of the model to capture mountain wave generation and propagation phenomena under complex topographic conditions. The computational domain spans 60 km in both horizontal dimensions with a vertical extent of 16 km. Adaptive mesh resolution dynamically ranges from 125 m to 10 km throughout the simulation domain, with mesh refinement actively guided by variations in velocity magnitude and potential temperature. The mesh consists of 554 394 elements and 103 635 nodes. A 3D bell-shaped mountain profile is mathematically described as follows: 4h(x,y)=h01+x2+y2a232 where h0 = 400 m represents the peak elevation and a = 1000 m denotes the characteristic half-width parameter. The stratified atmospheric background is characterized by N = 0.01 s-1, while the surface potential temperature initializes at θ0 = 293.15 K. The incoming flow maintains a constant velocity of u = (10,0,0)T ms-1. For numerical stability, an absorbing layer is implemented in the upper atmospheric region (10–16 km-altitude), with additional dissipative layers extending for 10 km inward from all lateral boundaries (comprehensive specifications available in Li et al., 2021).

In scientific applications demanding high-precision floating-point operations, microscopic rounding differences between CPU and GPU architectures are inevitable. In a fully dynamic simulation, these bit-wise differences can cause the Adaptive Mesh Refinement (AMR) algorithm to split elements differently over time. Comparing results across divergent meshes necessitates spatial interpolation, which introduces algorithmic noise and obscures the underlying arithmetic consistency. To eliminate structural variations and rigorously validate the GPU implementation, we conduct evaluations under two complementary fixed-mesh scenarios.

First, to validate the core operators during the non-stationary initial stage, we extract the refined unstructured mesh generated at t = 3000 s from an adaptive run and employ it as a fixed, static grid right from the beginning of the simulation (t=0). Both the CPU and GPU versions simulate the transient phase up to t = 3000 s with the AMR module deactivated. While this static mesh is not dynamically optimized for the early stages, it ensures an identical degree-of-freedom layout from t=0 and completely avoids spatial interpolation errors. Second, to verify long-term stability in the steady-state regime, the simulation is run up to t = 3000 s under the standard CPU-AMR configuration until the wave profile stabilizes. The mesh is then fixed, and both versions advance the physical time independently from t = 3000 s to t = 5000 s. The computational results for both scenarios are presented in Fig. 4. Figure 4a illustrates the 3D wireframe and a 2D cross-sectional slice of the mesh, demonstrating the localized high-resolution refinement adapted to the mountain wave. For the transient fixed-mesh run (Fig. 4b and c), the absolute differences between the CPU and GPU fields at t = 3000 s peak at 1.44 × 10⁻¹⁴ for the vertical velocity (w) and 4.73 × 10⁻¹⁵ for the potential temperature perturbation (θ′). For the steady-state continuation run (Fig. 4d and e), the differences remain bounded within a similar magnitude at t = 5000 s. These minuscule variations strictly correspond to the double-precision machine epsilon, confirming that the GPU-accelerated operators are fundamentally and mathematically consistent with the original CPU baseline across all simulation phases.