Coupling technologies for Earth System Modelling

This paper presents a review of the software currently used in climate modelling in


Conclusions References
Tables Figures

Introduction
Model coupling is essential for realizing multi-physics simulations based on two or more computing applications.An Earth System Model (ESM) is a quintessential example of a coupled model, which involves several interacting components simulating the atmosphere, oceans, land, and sea ice.The software that links together these model components is called a "coupler".Although their implementations differ vastly, couplers used in the geophysical community typically carry out similar functions such as managing data transfer between two or more components, interpolating the coupling data between different grids, and coordinating the execution of the constituent models.In general, coupling data must be regridded and passed between the components subject to different constraints such as conservation of physical quantities, stability of the flux exchange numerics, consistency with physical processes occurring near the component surface, etc.In addition, computational efficiency of the coupling on parallel hardware is of course required.This paper provides a review and a short comparative analysis of the main coupling technologies currently used in Earth System Modelling.Introduction

Conclusions References
Tables Figures

Back Close
Full The Earth System Modeling Framework (ESMF, http://www.earthsystemmodeling.org) is open source software for building modeling components and coupling them together to form weather, climate, coastal, and other applications.It is used and managed by a consortium of US agencies.
ESMF is comprised of a superstructure of coupling tools and component wrappers with standard interfaces, and an infrastructure of utilities for common functions, including calendar management, message logging, grid transformations, and data communications (Hill et al., 2004).Infrastructure utilities, including a tool for generation of interpolation weights, can be used independently from the superstructure.
ESMF offers two kinds of component wrapper: a gridded component which is associated with a physical domain, and a coupler component for transforming and transferring data between gridded components.ESMF components exchange information with other components through state objects, which contain representations of physical fields.
ESMF enables components to run sequentially, concurrently, or in mixed mode.Components can be nested.Applications usually run with all components linked into a single executable program, but there is also support for running separate components as multiple executables, and many infrastructure methods can be used in either context.ESMF can couple components written in Fortran or C. Its component wrappers may be layered on top of other coupling technologies (e.g.MCT, FMS).
ESMF supports a wide variety of grids and remapping options.Generation of interpolation weights and their application is fully parallel.ESMF supports first order conservative, bilinear, and a higher-order finite element-based patch recovery method for remapping in 2-D and in some cases 3-D.Logically rectangular and unstructured grids are both supported.There is a range of options with respect to masking and handling of poles and unmapped points.The remapping system is flexible and modular; Introduction

Conclusions References
Tables Figures

Back Close
Full the calculation of interpolation weights can be performed either during a model run or offline, and the application of weights can be made as a separate call.
Metadata is an important aspect of model documentation and interoperability.Methods of the ESMF Attribute class can be used to store, aggregate and output model metadata.Metadata is organized into packages, following community conventions such as the Climate and Forecast conventions (see http://cf-pcmdi.llnl.gov),ISO standards, and the METAFOR Common Information Model (Lawrence et al., 2012).Metadata packages can also be customized.
In order to adopt ESMF, modelers arrange their code as a set of gridded components and coupler components, and then split these components into standard ESMF methods (initialize, run, and finalize, each of which may have multiple phases).The next steps are to wrap native model data structures with ESMF data structures, and then register components with the framework.
Timing results for a variety of codes show that the overhead of using ESMF components is typically negligible (< 3 % of runtime), and that key operations scale to tens of thousands of processors.Grid remapping and parallel communications are highly scalable and extensible to many new grid types.The framework is very robust and is exhaustively tested nightly on 24+ platforms using a suite of over 4000 tests and examples.
For CMIP5, higher order interpolation weights generated by the ESMF offline tool were used in the CCSM4/CESM1 model to significantly reduce interpolation noise when mapping wind stress from atmosphere to ocean, relative to the bilinear method used previously.an atmosphere model, an ocean model, a land surface model, and a sea ice model.In addition, a coupler (or driver) is used to exchange boundary data between the components and to coordinate the time evolution of the physical models.CCSM is used to understand the Earth's global climate system, to predict the effects of climate change, and to understand past climates.It is developed as a high performance computing application but is used on a wide variety of platforms.The Community Earth System Model (CESM) is an extension of CCSM that includes an additional land-ice component, a higher altitude atmosphere model option, land and ocean biogeochemistry capabilities, and an atmospheric chemistry model.
Prior to CCSM4, CCSM system components ran concurrently as separate executables on distinct hardware processors, and a separate coupler mediated communication, performed grid interpolation, and implicitly handled time integration.With the CCSM4 release in 2010, a completely new approach to coupling climate components was taken within CCSM (Craig et al., 2012).CCSM4 is a single executable implementation that contains a top-level driver and components coupled via standard init/run/finalize interfaces.Individual components in CCSM4 can be laid out on processors in relatively arbitrary ways such that components can be run on identical or Introduction

Conclusions References
Tables Figures

Back Close
Full independent hardware processors.The top-level driver that runs on all processors controls the processor layout and time sequencing of the components.A separate coupler component, which can run on a subset of all the processors, still exists in the system to regrid and/or merge coupling fields, and carry out other coupler functions.
Components in CCSM4 are parallelized with MPI and OpenMP.The CCSM4 driver/coupler uses Model Coupling Toolkit (MCT, see Sect. 6) datatypes and methods extensively, with mapping weights generated offline.A new parallel I/O (PIO) library is being used and offers improved I/O performance particularly in the area of memory scaling.
The new implementation improves performance because of greater flexibility in laying out components on hardware processors compared to the prior concurrent-only CCSM3 system.CCSM4 can run on a single processor without MPI but is also highly memory and performance scalable for runs at high resolution.The scaling of the CCSM4 coupler has been evaluated at different resolutions and on different hardware platforms on up to 10 000 processors.FLOP intensive kernels scale linearly across all processor counts, resolutions, and machines.Memory intensive operations scale linearly at lower processor counts, but the scaling flattens out at higher processor counts as the number of gridcells per processor decreases below a few hundred.Communication-dominated kernels tend to scale sub-linearly at lower processor counts, and scaling tends to drop off above about 1000 processors.Scaling performance for communication dominated kernels is highly dependent on the machine and resolution.Overall, the improvements in the memory and performance scaling capability of the CCSM4 coupler compared to CCSM3 are significant.
The model is now being run at global resolutions of around one tenth of a degree on tens of thousands of processors, and several thousand years worth of CMIP5 simulations have been carried out at varied resolutions and on many different hardware platforms.Introduction

Conclusions References
Tables Figures

Back Close
Full Component-based design of model codes supposes defining standard component interfaces (SCI).The Flexible Modeling System (FMS) coupler is a domain-specific SCI: it is written quite narrowly to support ESMs and recognizes only a few components: an atmosphere, an ocean surface -including the sea ice -, a land surface, and an ocean.Any other components inherit a grid from these, e.g.atmospheric physics and chemistry from the atmosphere; terrestrial biosphere, river and land ice from the land surface; marine biogeochemistry from the ocean.In the FMS SCI, there are "slots" for each of the specific components listed above.Components must be "wrapped" in FMS-specific data structures and procedure calls.
The FMS coupler is designed to address the question of how different components of the Earth system are discretized, each one making independent discretization choices appropriate to its particular physics.In an atmospheric model, vertical diffusion is generally treated implicitly and stability is enhanced by computing the flux at the surface implicitly along with the diffusive fluxes in the interior.Simultaneously, land or ocean surfaces with vanishingly small heat capacity should be allowed.Therefore, the vertical diffusion of temperature in a coupled atmosphere-land system may lead to a tridiagonal matrix inversion which can be solved relatively efficiently using an up-down sweep, with the particularity that some of the layers are in the atmosphere and others are in the land.Moreover, if the components are on independent grids, the key flux computation at the surface is a physical process that must be modeled on the finest possible grid.Thus, the "exchange grid" (Balaji et al., 2006) on which this computation is performed in FMS emerges as an independent component for modeling the surface boundary layer.
A grid is defined as a set of cells created by edges joining pairs of vertices defined in a discretization.Given two grids, an exchange grid is the set of cells defined by the union of all the vertices of the two parent grids.Quantities being transferred from one parent grid to the other are first interpolated onto the exchange grid and then averaged onto the receiving grid.The general procedure for solving the vertical diffusion is Introduction

Conclusions References
Tables Figures

Back Close
Full thus split into separate up and down steps.Vertically diffused quantities are partially solved in the atmosphere and then handed off to the exchange grid, where fluxes are computed.The land or ocean surface models recover the values from the exchange grid and continue the diffusion calculation and return values to the exchange grid.The computation is then completed in the up-sweep of the atmosphere.This features is key in the design of the FMS coupler.
The FMS coupled modeling system also includes a parallel ensemble adjustment Kalman filter for data assimilation (Zhang et al., 2005) in which ensemble members are treated as concurrent components.
The FMS coupler has been shown to scale up to O (10 000) processors with fast surface processes coupling every atmospheric timestep (typically ∼ 15 min) and slow processes coupling every ocean timestep (typically 1 h).FMS has been active for over a decade.Its feature list and its performance still remain state of the art.The versatility of FMS is seen in GFDL's approach to CMIP5.GFDL has submitted four streams of modeling results to CMIP5: these include the CM3 model (Donner et al., 2011), which includes interactive aerosol chemistry for control, historical and projection runs; two Earth System models ESM2M and ESM2G for the carbon cycle runs (Dunne et al, 2012); high-resolution "time-slice" experiments using the HiRAM-C180 and HiRAM-C360 models (Zhao and Held, 2012); and near-term prediction experiments using a sophisticated ensemble coupled data assimilation (ECDA) system (Zhang et al., 2007) and the CM2.1 model.All of the models are built using different combinations of choices of atmospheric dynamical cores, atmospheric physics packages, ocean models and the ECDA system, all of which are available as FMS components.Introduction

Conclusions References
Tables Figures

Back Close
Full The OASIS3 coupler (Valcke, 2006(Valcke, , 2012) ) is the direct evolution of these first developments.Since 2007, OASIS3 is developed and supported thanks to an active collaboration between CERFACS and the French "Centre National de la Recherche Scientifique" (CNRS).OASIS3 is written in Fortran and C. In a coupled system assembled with OASIS3, the coupler itself forms a separate executable that performs the regridding tasks.The component models remain separate executables with main characteristics, such as internal parallelisation or I/O, untouched with respect to their uncoupled mode.To interact with the other components through the coupler, the component models need to link to the OASIS3 coupling interface library.The coupling interface library API includes calls to receive and send the coupling fields usually implemented within the model timestep loop.The characteristics of the coupling exchanges, e.g. the corresponding target or source component of an exchange or the coupling frequency, are not explicitly defined in the model code but in an external configuration file written by the user.At run-time the coupling library and the coupler perform coupling exchanges according to the information contained in this file.For regridding, OASIS3 includes the SCRIP library (Jones, 1999).
For each coupling field exchange, the different parts of a coupling field sent by the source model processes are gathered by one coupler process which regrids the whole coupling field and distributes it to the target model processes.OASIS3 can therefore be parallelised on a field-per-field basis in the sense that each coupler process can treat a subset of coupling fields.
OASIS3 success up to now can be explained by its great flexibility and its low intrusiveness in the component codes.OASIS3 is used today by about 35 different climate modelling groups in Europe, Australia, Asia and North America.In particular, OASIS3 is the coupling software used in 5 of the 7 European ESMs participating to CMIP5, i.e.CNRM-CM5 (Voldoire et al., 2011), IPSL-CM5 (Dufresne et al., 2012), CMCC-ESM (Vichi et al., 2011;Scoccimarro et al., 2011) high-resolution (∼ 2/3 • ) configurations but its limited parallelism will eventually become a bottleneck in the coupled simulation.Within the framework of the current EU FP7 IS-ENES (see https://is.enes.org)project, work continues to parallelise and extend the existing functionality and to establish comprehensive services around OASIS3 (see http://oasis.enes.org).

The Model Coupling Toolkit
The Model Coupling Toolkit, MCT (Larson et al., 2005;Jacob et al., 2005) embodies an application-neutral approach for creating coupled multi-physics models.Thus, MCT can be used in diverse scientific fields.Because MCT imposes no architecture on the application, the developer can freely choose the number of executables and model process composition.An API also allows the developer to choose such elements as coupling data description, parallel data movement, and support for parallel data transformation and interpolation.MCT design philosophy is that flexibility and minimal invasiveness are vital to the development of long-life-cycle coupled models.
MCT provides a Fortran-based object model for coupling construction and bindings for C++ and Python have been developed.An MCT datatype describes the coupled system's processor layout.MCT stores coupling field data in an object that supports arbitrary numbers of real-and integer-valued fields, indexed using string tokens.A domain decomposition descriptor (DDD) object uses virtual linearization to represent multidimensional index spaces.Parallel communication schedules are computed automatically from source and destination DDDs.Parallel data transfer is accomplished by calling paired send/receive methods with data storage and communication schedule datatypes as inputs.MCT provides distributed storage for precomputed interpolation coefficients from which it derives communication schedules for parallel interpolation.MCT assumes MPI-based parallelism but includes a small MPI-replacement library for non-parallel applications.

GMDD Introduction Conclusions References
Tables Figures

Back Close
Full MCT is highly portable and uses a GNU autoconf-based build system.MCT programming model derives from Fortran90: module use to access MCT classes and methods, declaration of variables of MCT datatypes, and invocation of MCT methods to perform coupling operations.To use MCT, the developer locates logical interaction points in the subsystem models, adds code to declare and initialize MCT datatypes for coupling, and inserts handshaking calls between model pairs to initialize communication schedules.Within the model "run" method, the user inserts calls to load the coupling data into MCT datatypes and calls MCT parallel communication and interpolation methods.
The biggest challenge in using MCT is defining virtual linearizations of mesh and index spaces.Most new MCT users, however, quickly build their own parallel coupled models after experimenting with the examples provided.Ease of use is MCT's primary benefit.Its limitations are lack of support for computation of interpolation weights and for MPI communicator construction.MCT has been the basis of all CCSM couplers since 2004, supporting thousands of model-years of coupled climate simulation.In particular, the MCT-based coupler, cpl7, is being used for all of the CMIP5 integrations being performed using CCSM4 and CESM1 (see Sect. 3).
Exascale platforms will require refactoring key MCT portions.Paucity of per-core memory at exascale requires reexamination of field data copying and DDD replication.Employing compatible mesh representation software in all components could eliminate field data copying.Employing space-filling curves as compact virtual linearizations could minimize DDD replication costs.Tolerating hardware faults and dynamic load balance mean MCT assumption of a static processor pool must be revisited.Work is under way to implement coupling in the presence of dynamically-varying processor pools.Currently, MCT supports applications on tens of thousands of processors and is well positioned for future coupled model challenges.

Conclusions References
Tables Figures

Back Close
Full The Bespoke Framework Generator (Ford et al., 2006;Armstrong et al., 2009) (BFG, http://www.cs.manchester.ac.uk/cnc/projects/bfg) owes its development to an analysis of the Met Office future coupling requirements, the requirements of the GENIE paleoclimate coupled model (Armstrong et al., 2009, http://www.genie.ac.uk) and Community Integrated Assessment System (CIAS, http://www.tyndall.ac.uk/research/cias, Warren et al., 2008).BFG allows the user to choose the underlying coupling technology, taking coupling metadata as input and generating tailored (bespoke) wrapper code to be compiled and linked with the user's code and the chosen coupling library.Separating the coupler from the science code offers an additional layer of flexibility which can improve portability, performance and maintainability, thus future-proofing the code.BFG treats transformations (such as grid transformations) in the same way as model code.Intrinsic transformations (such as those in OASIS3) can also be supported.BFG has been designed to be generic and extensible; thus BFG may be used in application domains other than ESM (Warren et al., 2008;Delgado-Buscalioni et al., 2005).
BFG2, the current implementation, can be run directly from the BFG portal (see http://bfg.cs.man.ac.uk).BFG2 supports models written in Fortran as a module containing subroutines or a set of subroutines, C as a set of procedures and Python as a class with a set of methods.The coupled model behaviour is specified as composition and deployment metadata in XML.At the language level, BFG2 supports passing data to and from subroutines/procedures/methods via arguments and/or in-place put/get calls.The former approach is similar to that used by ESMF, CPL7 and FMS (see Sects. 2, 3 and 4) and the latter to MPI, OASIS3 and CCSM3 (see Sects. 5 and 3).Since models are not main programs, BFG2 is able to map models for deployment either as a single executable or as multiple executables, each containing one or more models.Coupling connections can, on first use, be initialised in a variety of ways, including from a model or a file.Coupling technologies currently supported to exchange data between models include argument passing, MPI or OASIS4 (Redler et al., 2010).The OASIS4 BFG

Conclusions References
Tables Figures

Back Close
Full implementation supports the specification of grids using an XML representation of the Gridspec (Balaji et al., 2006) as well as the use of intrinsic OASIS4 transformations.BFG2 requires model (science) code to conform to some simple coding rules and to be described by definition metadata.Composition metadata specifies how the models (and transformations) are connected together and deployment metadata specifies how to map the models onto the available hardware and software resources.BFG2 takes the metadata as input and generates bespoke control and communication code using a Python program.
In conclusion, BFG isolates science code from the coupling infrastructure and provides a metadata-driven code generation system to provide flexibility in model composition and deployment.This flexibility allows BFG to achieve similar performance to hand-written code and provides the opportunity for finer-grain coupling than is typical today.BFG is currently used within CIAS, where it is used to couple over 20 different Integrated Assessment model configurations.BFG is not used in any CMIP5 runs but offers a potential solution for the coupling of future Earth System Models.Some limitations of BFG2 are, in particular, that wrapper code must be regenerated whenever the metadata changes and that data partition metadata for use with parallel models is not yet supported (but will be added soon, using MCT for MPI implementations).BFG2 is being extended to support models written in a less modular wayin particular, model codes which are main programs, codes with internal control, and models where the source code is not available.In the future, BFG2 will be extended to support ESMF and CPL7 as coupler "targets".Finally, we will investigate the feasibility of using BFG2 to couple models that conform to other frameworks by generating appropriate adaptor code.Introduction

Conclusions References
Tables Figures

Back Close
Full limited to the likely outcomes of the different design strategies.While the details of the approaches vary, features of the different coupling technologies typically include an ability to communicate data between components, regrid data, and manage the time evolution of the model integration.Coupling using a concurrent multiple executable approach (e.g.OASIS3) requires minimal modification to existing component codes but limits the ways they can be mapped to hardware, which can hinder performance.Coupling via component-level interfaces within one integrated application (e.g.ESMF, CPL7 or FMS) generally requires users to split components into initialize, run, and finalize methods, and may limit the places where data exchanges can happen.Although this can simplify program flow, it can also affect time sequencing and require scientific reformulation; however, because components can be run sequentially or concurrently, there are additional opportunities for performance optimization.This "integrated" approach also enables components to be nested, with multiple coupling levels.Coupling toolkits (e.g.MCT) are designed for a la carte use of classes and methods.They allow great flexibility for building custom parallel coupling mechanisms, with either single or multiple executable approaches.Subsets of other coupling technologies (e.g.ESMF utilities) can also be used separately to solve some coupling problems.Research in generative programming (e.g.BFG) explores potential ways to unify the different coupling approaches.In the end, science needs both flexible coupling capabilities and high performance.Both have be- as for other software, both finding additional opportunities for parallelism and improving communication mechanisms to better overlap communication with computation.Over much of the past two decades, several groups have worked relatively independently to develop Earth System coupling technology.In many aspects, those implementations have converged as individuals gain experience and as common science and high performance requirements drive implementations.At the same time, different scientific communities continue to benefit from fundamentally different solutions.Moving forward, the community recognizes the potential benefit of much closer collaboration especially considering the uncertainty of future hardware.In addition, there is recognition that if future hardware requires significant rewrites of Earth System models in new programming languages, an opportunity might present itself to unify coupling approaches and share developmental costs.Introduction

Conclusions References
Tables Figures

Back Close
Full Screen / Esc Printer-friendly Version Interactive Discussion Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Screen / Esc Printer-friendly Version Interactive Discussion Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Screen / Esc Printer-friendly Version Interactive Discussion Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | 4 The GFDL Flexible Modeling System Screen / Esc Printer-friendly Version Interactive Discussion Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Screen / Esc Printer-friendly Version Interactive Discussion Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | , EC-Earth (Hazelger et al., 2011), and MPI-ESM from the Max Planck Institute.OASIS3 is also used successfully in few relatively Discussion Paper | Discussion Paper | Discussion Paper | Screen / Esc Printer-friendly Version Interactive Discussion Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Screen / Esc Printer-friendly Version Interactive Discussion Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Screen / Esc Printer-friendly Version Interactive Discussion Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | come crucial in the last few years as coupling complexity and resolution have rapidly increased and these trends are expected to continue in the future.Continual improvement in coupled climate model performance may become more difficult.Most of the gains in the last decade came from faster hardware on a per processor basis and improvements in grid decompositions, memory parallelization, and communication algorithms.Unfortunately, future generation hardware is likely to consist of orders of magnitude more processors that are slower, heterogeneous, and with less and slower memory.Moving into the exascale era will require, for coupling technology Discussion Paper | Discussion Paper | Discussion Paper | Screen / Esc Printer-friendly Version Interactive Discussion Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper |

3 The new CPL7 coupler designed for CCSM4 and CESM1
Newer versions of CESM use weights generated by the ESMF tool throughout the model for reasons of speed, accuracy, and ability to handle many types of grids.ESMF component interfaces are now supported in CESM as well.ESMF re-