Climate and weather modeling have historically been among the most computationally demanding domains using high-performance computing (HPC). Its history parallels that of modern computing itself, starting with experiments on the ENIAC and continuing through several changes in supercomputing architecture, including the vector and parallel eras.

The transition from vector to parallel computing was disruptive, to use a currently popular term. The computing industry itself was transitioning from being primarily responsive to military and scientific needs, to being dominated by a mass-market demanding cheap and ubiquitous access to computing. This gradually led to the demise of specialized computational machines and the high end of the market also being dominated by clusters built out of mass-market commodity parts .

The community weathered the challenge well, without significant loss of pace of scientific advance. More narrowly stated, Earth system models (ESMs) continued to demonstrate continual increases in both resolution and complexity across the vector–parallel transition. For example, the typical resolution of climate models used for the Intergovernmental Panel on Climate Change (IPCC) assessments and their complexity (the number of feedbacks and phenomena simulated), exhibits a steady increase from the 1990 First Assessment Report (known as FAR) to the Fourth Assessment Report (AR4) see e.g., the iconic Figs. 1.2

https://www.ipcc.ch/publications_and_data/ar4/wg1/en/figure-1-2.html

A second disruption is upon us in the current era. Current computing technologies are based on increased concurrency of arithmetic and logic, while the speed of computation and memory access itself has stalled. This is driven by many technological constraints, not least of which is the energy budget of computing . These massive increases in concurrency pose challenges for HPC applications: an era where existing applications would run faster with little or no effort, simply by upgrading to newer hardware, has ended. Substantial recoding and re-architecture of applications is needed. This poses particular challenges to applications such as climate modeling, where we must simulate many interacting subsystems. The state of play of climate computing in the face of these challenges is surveyed in and references therein: for the current generation of technology, the gains to be had seem modest, and the effort of recoding immense. Whether we will continue to demonstrate continued increases in resolution and complexity through this transition remains to be seen.

ESMs

Note that we are using the term ESM generically to denote any level in the hierarchy of complexity of weather and climate models: ranging from single-component models, e.g., an atmospheric general circulation model, to models that include coupling with the ocean and land, biospheres, an interactive carbon cycle, and so on. See Fig. .

Multi-physics codes are particularly challenged by the coming trends in HPC architecture. These codes typically involve many physical–chemical–biological variables (complexity) and associated process representations in code. Computational load is evenly distributed across many components, each embodying different physics: there are no performance hotspots. This also means that fresh operators and operands – embodied in physics subroutines and associated variables – are constantly being transferred to and from memory with each context switch, and locality and reuse are hard to achieve. This is particularly unsuited to the novel architectures currently on the horizon. These include Graphical Processing Units (GPUs), which can concurrently process O(100) data streams following the same instructions sequence, and the Many Integrated Core (MIC) architecture, which allows for O(100) execution threads to access the same memory. These hardware trends have made the cost of data movement prohibitive relative to computing itself; thus, strongly favoring codes where both instructions and data have a high rate of reuse and computational intensity (ratio of floating-point operations to memory operations). Algorithms that exhibit fine-grained concurrency, where multiple computationally intensive and concurrent data streams follow the same instruction sequence, are best adapted to the emerging architectures of this decade.

The next computational landmark at this juncture is the exascale, O(1018) operations per second. Given the earlier discussion on the stalling of Moore's Law, the rate of atomic arithmetic operations is still O(109) per second, thus requiring us to achieve O(109) concurrency. While continuing to extend physical and dynamical algorithms toward the fine-grained concurrency of the coming era, we believe multi-physics codes must also attempt to share the available concurrency across many physical components, in order to best exploit these new systems. Of the many factors of 10 increase in performance needed to get to the exascale, we believe at least one can come from component organization. We propose here a major architectural change in the construction of coupled models, such as ESMs. We demonstrate here a possible approach to extending the current rather modest amount of concurrency among ESM components (typically 2–4 top-level realms such as atmosphere, ocean, and land) to a more massive increase in coarse-grained component concurrency (CCC).

In this study, we examine the radiation component (which computes radiative transfer in the atmosphere in response to dynamically evolving concentrations of radiatively active chemical species) of the Geophysical Fluid Dynamics Laboratory (GFDL) Flexible Modeling System (FMS). This is a relatively expensive component of the atmospheric physics, and is usually run at a much coarser time step than the rest of the atmosphere (temporal subsampling), purely for expediency rather than any physical justification. Other approaches to reducing the computational burden of radiative transfer include subsampling in the spectral domain or in the spatial as well as the temporal domain . Some of these methods have been shown to be effective over short timescales (e.g., numerical weather prediction and medium-range forecasting) but contribute to model bias over climate timescales. Adaptive methods that tune the subsampling by examining the degree of spatial and temporal correlation in model fields have also been proposed .

We focus here on temporal subsampling. This purely expedient choice of time step has been shown by to be a potential source of instability and bias in radiating atmospheres. also showed that this problem gets considerably worse as the resolution of models increases. A useful way to think about this is that using different time steps for the radiation component, vis-à-vis the rest of the physics, creates a discrepancy between the cloud field and the cloud shadow field seen by the radiation component, which can lead to numerical issues. Our method permits us to reduce the time step to match the rest of the atmosphere, with the same time to solution, at a modest computational cost in terms of allocated processors. This method does not rule out subsampling along other dimensions (spatial or spectral), which may be superimposed as well in future developments. The effects of subsampling are not fully understood yet, and further study is needed to understand how results converge as various forms of subsampling are eliminated. That said, subsampling is clearly a matter of expediency and reducing computational expense: there is no case at all to be made that it is in any way numerically or physically superior to the alternative.

The structure of the paper is as follows. In Sect.  we briefly review current approaches to parallelism in ESMs, particularly in the coupling framework. In Sect.  we describe our approach to coarse-grained concurrency, how it is achieved without increasing data movement. In Sect.  we show results from standard AMIP the Atmospheric Model Intercomparison Project; simulations using the CCC approach. The associated computational results show decreased time to solution for concurrent vs. serial approaches in otherwise identical physical formulations, and the ability to run with a much smaller radiation time step without increasing the time to solution. Finally, in Sect.  we discuss plans and prospects for extending this approach further within FMS, and its potential application on novel architectures.

Weather and climate modeling has always been in the innovative vanguard of computing, dating all the way back to the origins of modern computing in John von Neumann's pioneering studies . The ability to apply instruction sequences to multiple data streams – concurrency – has long been a cornerstone of performance engineering. The pioneering vector processors of Seymour Cray's era in the late 1970s allowed a data stream to flow through a hardware innovation known as vector registers, which allowed for the same instruction sequences to apply to each succeeding element in the data stream, known as SIMD (single-instruction multiple data). Over time, vectors grew to support extremely complex programming sequences, evolving into single-program, multiple data, or SPMD.

In the 1980s, machines such as the Cray X-MP (the MP stood for multi-processor) were introduced. Here for the first time parallelism appears at a very high level, allowing for the concurrent execution of multiple tasks, which were themselves SPMD vector programs. This was the first introduction of coarse-grained concurrency. This led to the development of the MPMD (multiple program, multiple data) framework. Soon after, distributed computing, consisting of networked clusters of commodity computers, known as symmetric multi-processors, began to dominate HPC, owing to the sheer advantage of the volume of the mass market.

To take advantage of distributed computing, new techniques of concurrency began to be developed, such as domain decomposition. Here the globally discretized representation of physical space in a model component is divided into domains and assigned to different processors. Data dependencies between domains are resolved through underlying communication protocols, of which the message-passing interface MPI; has become the de facto standard. The details of message passing are often buried inside software frameworks (of which the GFDL FMS, described in , is an early example), and this convenience led to the rapid adoption of distributed computing across a wide variety of applications. Within the distributed domains, further fine-grained concurrency is achieved between processors sharing physical memory, with execution threads accessing the same memory locations, using protocols such as OpenMP .

Climate computing has achieved widespread success in the distributed computing era. Most ESMs in the world today are MPMD applications using a hybrid MPI-OpenMP programming model. At the highest end, ESMs or at least, individual components within them; see e.g.,S.-J. Lin and C. Kerr, personal communication, 2013 have been run on O(105)-distributed processors and O(10)-shared-memory execution threads, which places them among the most successful HPC applications in the world today . Even higher counts are reported on some leadership machines, but these are more demonstrations than production runs for science e.g.,.

Before we begin describing a method for casting the radiation code in FMS as a concurrent component, we need to describe the current methodology shown in Fig. . Concurrency between atmosphere and ocean component on the slow coupling time step is achieved using distributed computing techniques, with the components running on separate processor sets or PElists. In FMS terminology, a PE or processing element is a unit of hardware supporting a single execution thread, sometimes called a core. A PElist is synonymous with a communicator in MPI terminology, and lists the PEs assigned in distributed memory processing. Each PE in a PElist can spawn multiple shared-memory execution threads. These threads are assigned to other PEs to avoid contention. Coupling fields are transferred between atmosphere and ocean using the exchange grid and message passing. Within the atmosphere component, shared-memory parallelism using OpenMP is already implemented. For the dynamics phase, the OpenMP acts on individual loops, some of which may contain calls to subroutines or comprised of large programmatic constructs. These include regions where concurrency is on slabs (horizontally tightly coupled) and others organized in columns (vertically tightly coupled).

Unlike the dynamics, the physics is organized entirely columnwise, and individual columns – the k index in an (i,j,k) discretization – have no cross-dependency in (i,j) and can execute on concurrent fine-grained threads. The arrays here can be organized into groups of vertical columns, or blocks, that can be scheduled onto the same OpenMP threads at a high (coarse) level – meaning a single thread will persistently see the block (thus assuring thread data affinity) through the complete Atmos-down phase (sans dynamics), and again through the Atmos-up phase.

We now come to the reorganization of components for CCC in the current study. Understanding that the radiation is but one phase of the physics, which is already utilizing blocks and OpenMP threads, it makes sense to extend the concept and have the radiation run concurrently in a separate group of OpenMP threads. In the concurrent radiation architecture, shown in Fig. , the decision was made to utilize nested OpenMP, instead of a flat thread pool. Each MPI rank assigned to the atmosphere PElist starts up an OpenMP region with two threads, one to act as the master for the radiation and the other to drive the non-radiation components. These master threads are able to utilize the nested OpenMP constructs, to start up a number of atmosphere threads (A) and radiation threads (R), where the total numbers of threads T=A+R. For a given value of T, A and R can be dynamically adjusted during the course of a run to achieve optimal load balance. Because the radiation and atmosphere concurrency occur at a high level, the memory space is unique to each component and furthermore, the memory space is unique to each block. This separation of memory spaces ensures there are no performance deficiencies due to cache coherency effects (false sharing or cache invalidations). A single data synchronization point (copy) at the end of each atmospheric time step ensures that the atmospheric and radiation components remain completely independent.

In the limit, we could create blocks containing a single column, so that A and R both equal the number of columns in the domain, and T=2A. But the overheads associated with moving in and out of threaded regions of code must be amortized by having enough work per OpenMP thread instance. Current processors rely on having a moderate number of data elements to achieve best performance by hiding the various latencies for each instruction, including time spent waiting for operands to be loaded to memory registers. Empirically, on current technology, we have found optimal results using blocks of O(32) columns.

We are at a critical juncture in the evolution of high-performance computing (HPC), another disruptive moment. The era of decreasing time to solution at a fixed problem size, with little or no effort, is coming to an end. This is due to the ending of the conventional meaning of Moore's Law , and a future where hardware arithmetic and logic speeds stall, and further increases in computing capacity are in the form of increased concurrency. This comes in the form of heterogeneous computing architectures, where co-processors or accelerators such as Graphical Processing Units (GPUs) provide SIMD concurrency; the other prevalent approach is in the Many Integrated Core (MIC) architectures, with a vastly increased thread space, an order of magnitude higher than the O(10) thread parallelism achieved with today's hybrid MPI-OpenMP codes.

It is very likely that radical re-imagining of ESM codes will be necessary for the coming novel architectures . That survey of the current state of play in climate computing notes that multi-physics codes, which are fundamentally MPMD in nature, are particularly unsuited to these novel architectures. While individual components show some speedup, whole MPMD programs show only modest increases in performance see e.g.,. Other approaches, such as the use of inexact computing , are still in very early stages.

We have demonstrated a promising new approach for novel and heterogeneous architectures for MPMD codes such as Earth system models. It takes advantage of the component architecture of ESMs. While concurrency has been achieved at the very highest level of ESM architecture shown in Fig. , the components are themselves MPMD within a hierarchical component architecture.

In the light of our discussion we propose a precise definition of a component as a unit of concurrency. For the purposes of the coarse-grained concurrency (CCC) approach, a component may be defined as one of many units in a multi-physics model, which is itself SIMD for the most part. While the word has been loosely used earlier, this study has provided guidance on how we should think about components, and thus, this definition will be followed for the rest of the discussion in this section. Fine-grained parallelism approaches, such as those surveyed in , may be applied within a component as so defined, but are likely to fail above that level. A substantial increase in overall scalability of an ESM may be achieved if several components are run concurrently. We are currently exploring CCC in several other computationally burdensome model components, including atmospheric chemistry and ocean biogeochemistry. It is clear, however, that this is not an universal solution: given the constraint that concurrent components can only see each others' time-lagged state, some components are too tightly coupled to be amenable to the CCC approach.

Furthermore, we have demonstrated a method where multiple components that share a large number of model fields can be run concurrently in shared memory. This avoids the necessity of message passing between components that need to synchronize on fine timescales.

We believe the CCC approach will afford very tangible benefits on heterogeneous architectures such as GPUs, and architectures with a wide (O(100–1000)) thread space, such as MICs.

Threading within a SIMD component has not been shown to scale beyond a rather modest thread count. By running multiple components within a single thread space, the thread count can be considerably increased.

There remain caveats to this approach. As shown in the discussion of Eq. () above, the coupling of concurrent components might be formally unstable. We are exploring more advanced time-coupling algorithms, including three-time-level schemes such as Adams–Bashforth see. Such algorithms have been successfully used within the atmospheric dynamical core of FMS, for two-way nesting. In this approach, the coarse- and fine-mesh components execute concurrently rather than serially as in conventional nesting approaches . We are also exploring a combination of the three-time-level schemes with time-staggering of components, which no longer suffers from formal instability.

We conclude that coarse-grained concurrency remains a very promising road to the future of Earth system modeling on novel, massively concurrent HPC architectures.

Source code and data, including model output and performance data, associated with this study are freely available upon request.