Geoscientific Model Development Next generation framework for aquatic modeling of the Earth System

Next generation framework for aquatic modeling of the Earth System B. M. Fekete, W. M. Wollheim, D. Wisser, and C. J. Vörösmarty CUNY Environmental Cross-roads Initiative, The City College of New York at the City University of New York, 160 Convent Avenue, New York, New York, 10031, USA Complex Systems Research Center, Institute for the Study of Earth, Oceans and Space, University of New Hampshire, 39 College Rd., Durham, 03824, USA also at: NOAA-Cooperative Remote Sensing Science and Technology Center, The City College of New York at the City University of New York, 160 Convent Avenue, New York, New York, 10031, USA Received: 24 February 2009 – Accepted: 4 March 2009 – Published: 17 March 2009 Correspondence to: B. M. Fekete (bfekete@ccny.cuny.edu) Published by Copernicus Publications on behalf of the European Geosciences Union.


Introduction
Earth System modeling is an increasingly challenging task due to the rapidly increasing complexity of the models themselves (Hall and O'Connell, 2007).Modern land surface, ecosystem, hydrological, etc. models represent biogeochemical and physical processes in such detail that model codes become very complicated.In addition to the complexity of the models themselves, the IT infrastructure upon which the models are built is growing more complex as well.While both computational power and mass storage media capacity have undergone exponential growth following Moore's law, the speed of accessing the mass storage devices clearly lagged behind.As a consequence, applications today are often limited by the read and write access of the vast amount of data required by the models.One way to reduce the limitations of slow storage devices is to trade data storage for processing on the fly.For instance, downscaling algorithms built into models can reduce the amount of data that the models need to read.However, this solution comes at the price of performing computationally intensive interpolation algorithms as part of the model simulations.Such an approach essentially makes the already complicated models even more complex.
To complicate the matter further, the efficient management of the vast amount of data available to Earth Scientists requires sophisticated solutions.The era of "ad hoc" input/output using primitive, home-grown data formats is over; scientific data need a common management framework that provides catalogs, documentation, discovery mechanisms and preprocessing to feed the scientific applications.While emerging new data management methods and tools ultimately help the end-users, they put extra burden on the model developers to enable increasingly complex data services in Figures Back Close

Full Screen / Esc
Printer-friendly Version Interactive Discussion already complex models.When one considers that the computational power of single CPU core has remained more or less steady over the last four years while the Moore's law was realized by adding more cores to the CPUs (dual and quad core CPUs becoming common even in ordinary desktop computers) the need for implementing Earth System model components in a parallel processing environment is inevitable.Furthermore, parallel computers have vastly different architectures ranging from symmetric memory processing (SMP, when multiple processors have access to the same memory), non-uniform memory access (NUMA, when processors have access to faster memory individually and share common access to slower memory units) and cluster computers (e.g.Beowulf, when entire computers are clustered via fast network connections).These different platforms need very different parallelization strategies.
Computer science has delivered numerous essential components to make the utilization of these different architectures easier.For instance, Message Passing Interface (MPI) is commonly used in many parallel applications.No new development platform has emerged yet that could help to automate the parallelization of scientific tasks.When parallel computers became commonplace more than twenty years ago, one would have thought that new "non-procedural" languages (like LISP or Prolog) would emerge allowing the description of the computational tasks in a conceptual manner that intelligent interpreters or compilers could analyze and optimize to execute on parallel CPU systems.The last two or three decades brought numerous new programming languages into play (C++, Java, Python) and new concepts like structured and object oriented programming.The new languages primarily aided the construction of complex applications by compartmentalizing the application development and allowing higher levels of modularity, but these languages remained "procedural" in the sense that the code describes the tasks as a sequence without giving hints to the compiler or interpreter, what can be executed independently, what needs to precede other steps.Some languages incorporated complex event handling capabilities primarily to support the development of modern graphical user interfaces but the event handling concepts are rarely suitable for Earth System models.
The lack of new development concepts might be due to the large differences in them demands of specific applications, so no common solution has yet emerged.As a consequence, we argue that Earth Scientists need to develop their own model development platform that is designed to address Earth Science problems.This new model development platform needs to be built on a new paradigm that allows the description of scientific tasks in a hardware platform independent manner.

Existing modeling frameworks
The need for unified modeling frameworks was realized long time ago and numerous solutions are emerging with different design goals.While all modeling frameworks are aimed at simplifying the integration of model components one way to categorize them might be the level at which the model integration occurs.The Flexible Modeling System1 (FMS, developed at the Geophysical Fluid Dynamics Laboratory, GFDL) and the Earth System Modeling Framework2 (ESMF, which might be viewed as descendant of FMS, was originally initiated by NASA, and currently under development at NCAR) both provide a series of utilities and services to model developers in the form of function libraries that provide solutions to various computational demands.ESMF offers function library tools for handling components (primarily grid) and associated arrays of variables, partitioning of components for execution on multiple processors, distribution of data to processors, exchange of data between processors and the gathering calculation results, management of time stepping, data exchanges between different components (coupling), input/output and log facilities.ESMF can be used as a primary mean for managing space and time in an Earth System model or it can be used as a wrapper around existing models that otherwise have their entirely different internal structure.While ESMF can be used as a "glue" between different models, its primary goal is to provide unified internal model infrastructure.In this respect, ESMF is a huge step forward as a new hardware-independent modeling platform, but the use of ESMF still has steep learning curve and requires the model developer to deal with low-level details (e.g.partitioning the model domains into computational subregions, managing the data distribution between compute nodes, etc.).
Open Modeling Interface 3 (OpenMI) funded by the European Commission and the Common Component Architecture 4 (CCA) initiated by the U.S. Department Energy provide solutions to higher level model coupling.Both OpenMI and CCA are designed to couple entire models (just like when ESMF is used in a "glue wrapper" mode.The capability to couple entire models is certainly useful, particularly the wrapping existing models into a larger modeling infrastructure, but at some point the legacy models will have to be retired as computer hardware and the software infrastructure makes them obsolete. Perhaps the biggest shortcoming of coupling entire models is that as soon as these models maintain their own way of managing space and time (and consequently perform input/output in their own way), the necessary wrapper environment to handle the model differences needs to be fairly complicated.Furthermore, space and time management are exactly the most critical elements for parallel execution, so once they are embedded deeply in the model, any support to aid parallelization, and computational load balancing from the modeling framework standpoint becomes difficult if not impossible.
A third set of popular modeling environments came from commercial software vendors such as Matlab  (Muetzelfeld and Masshender, 2003) and COMSOL9 .These are probably the tools that are the closest to the new model development platforms that Earth system scientists will need.Actually, numerous Earth System model components have implementation using these tools (e.g.Topmodel written in Matlab, etc.).The common service that these tools provide is a simple means to perform computation with different levels of complexity on variable arrays, so that developers could focus on the actual computational tasks and let the software to perform rudimentary data management.While these software development platforms are often extremely valuable for quick prototyping, they don't scale well to the massive computational tasks of large Earth System models.Furthermore, these software tools are rarely geared to Earth science needs and due to the closed source nature of commercial software, adding the needed extensions is rarely feasible.

Common Earth System model function
The most important common functions that Earth System models and model components share is the discretization of space and time, linkages between different computational domains that discretize space and time differently (coupling) and calculations on the discrete spatial objects at some computational frequency.The discretization of time is perhaps more trivial since it is only a one dimensional problem and the major challenge is to map the regular time intervals to the somewhat irregular time increments in traditional calendars (e.g.varying length of month, leap years, the disconnect between weekly and monthly increments).A discretization of space is more challenging.Numerous solutions to numerically represent spatial features have emerged over the last few decades.The three characteristically different approaches are Geographical Information Systems, remote sensing image processing systems and Earth System Model tools.The primary focus of traditional GIS was to represent static spatial features via series of contours and polygons.Representing similar features in a gridded Introduction

Conclusions References
Tables Figures

Back Close
Full Screen / Esc Printer-friendly Version Interactive Discussion context was added later as a mean to make overlay calculations easier.The line between gridded GIS and image processing is blurry, but the biggest difference perhaps is that image processing tools need to manage not only static images, but the also changes that occur over time, yet typically at low temporal frequency (often in the order of daily to bi-weekly intervals).Earth system model tools (which don't really have a name on their own, but can be regarded as the tools supporting NetCDF10 developed mostly by NCAR) were designed to manage space and time at high temporal frequencies by compromising on spatial resolution.These tools treat three or four dimensional arrays (representing 2D or 3D domains in time) as single objects.
The synergy of the GIS, image processing and Earth system tools is critical for the next generation of Earth System models to enable modelers to perform simulations over the computational domains that are most suitable for the particular model components and yet enable model components (operating on one type of spatial domain layout) to interface with other components.This synergy will require a generalized means of exchanging information between different domains (coupling). for applications where novel approaches representing both is integral part of the science (e.g.adaptive finite difference meshes) as these technologies mature they could be implemented in the modeling frameworks.

Next generation framework for aquatic modeling of the Earth System
We have developed several modeling frameworks (Data, Assembler, FrAMES, and NextFrAMES) in the last 15 years following the previously described design goals.
The latest effort building on the Framework for Aquatic Modeling of the Earth System (FrAMES) (Wollheim et al., 2008) was started about a year ago in response to the need identified in the ongoing NASA WaterNet Solution Networks and IDS projects.While FrAMES (as its name implies) was primarily designed to support hydrological and constituent transport and processing models, NextFrAMES eliminates most of its limitations and offer a modeling platform for wide range of land surface modeling applications.2.1 NextFrAMES model structure NextFrAMES modeling XML (document type definition provided in Appendix A) organizes models around components (aggregate, container, input, and region (Fig. 2).

NextFrAMES includes
Components represent spatial domains as a vector of computational objects.The actual handling of the physical domains and the translation from the domain elements (discrete points, grid cells, etc.) to the vector of objects is delegated to I/O plugin infrastructure.Components maintain lists of variables as a vectors (of the same length as the number of computation objects in the component).Components can contain any number of variables and parameters.Parameters are constant values through space and time.
While components themselves have no information about the actual layout (e.g.discrete points, grid, gridded networks) of its computational objects, it is the I/O plugin interface that provides the necessary object topologies.An object topology describes the spatial relationships between computational object.For instance, a 2D grid domain could define horizontal and vertical neighbors (2D topology).Gridded river networks (Fekete et al., 2001;V ör ösmarty et al., 2000) could define downstream and upstream river reach (tree topology).Components could have any number of topologies and the available topologies determine the available spatial operators (derivative, route) the can be requested.Each component has its own temporal frequency at which the component operates.
Components can form a hierarchical tree through container components.Container components hold other components, interfaces and processing modules.Container components inherit their vector of computational objects form other components.Currently only input components can define new vector of computational object.
Aggregate components (Example 2) provide a special means to perform aggregate functions (mean, minimum, maximum, standard deviation, percentage) over a variable of source component and map the result to the vector of computational objects in the aggregate component.For instance, aggregate components can be used to to another through interfaces and hold the modules to perform calculation tasks.An interface maps a variable from a source component to a new variable in the container component via coupler.The coupler is a simple weighting mechanism (established by the run-time engine) assigning a set of object from the source component and the corresponding weighting to a particular object in the destination component (Fig. 2).Couplers are one directional, but separate couplers from components can be established in each direction.Establishing couplers is hidden to the modeler and it is NextFrAMES' duty to find out how to couple two different set of computational objects.
The main purpose of container components to perform computations on the variables within the container via process modules provided in dynamically linked plugin record.The execute functions responsibility is to read the passed variable values and return the results in the output variable elements of the user defined record.
The variables and parameters of a process module are associated to variables and parameters defined in the container components via aliases, therefore the same process module can be reused within the same models for different variables and parameters.
Simplified mechanism to define modules is provided through equations allowing the symbolic definition of calculation within the modeling XML (Example 5).The equation module uses alias mechanisms to relate symbolic variables to variables and parameters of the container component.
Parameters (that are uniform in space and time during model calculations) can be defined at any level of the components hierarchy down to the module level, which determine the scope of the parameters.For instance, parameters defined in the "root" container (which is the model clause of the modeling XML) have global scope (i.e.known throughout the whole model).Parameters defined in descendent container component are only known within the container and in any of it descendent containers.Parameters defined in modules are only known in the module where they were defined.
Derivative and route operator modules (Example 6) perform calculations across topologically linked computational objects.Derivatives calculate forward, backward and center differences in one of the axis direction on 2D or 3D topology.Route modules perform upstream accumulation on objects with tree topology (river network).
Input variables in any module (process, equation, derivative or route) have to be defined before the module is specified.Normally, variables are either defined through interfaces or as outputs from previous modules.Variables that are updated in subsequent module calculations can be defined as initial variable.Uniform initial value of the initial variables can be provided in the model XML or spatially varying values from previous model runs can be loaded through "restart" or "state" files defined as container attributes.The purpose of input components (Example 7) is to translate variables from input sources the vector of computational objects (Fig. 2).An input component uses an I/O plugin to access the content of the input source and retrieve optional object topologies.The I/O plugin is specified in the modeling XML along with the source URL of the external data.Input components can inherit the time step from the input source our specify the input frequency via the "time step" attribute.When the requested frequency is slower than the time step of the input data, the input component performs temporal averaging for the requested frequency.When the input data has lower frequency than the requested time step, the same values are returned at each high frequency data request as a step function.The "offset" attribute (not shown in example) can be used to request low frequency input records in advance or delayed.The offset capabilities will allow the implementation of advanced downscaling (beyond the simple step function) such as bilinear, spline, etc interpolation.
Region components provide a simple means to subset the vector of computational objects, based on simple criteria on a single variable from the source components.
For instance, one could implement a plant growth model in NextFrAMES and define bare soil regions (that would vary over time) as a function of vegetation state.Regions actually inherit the layout of the parent component but allow masking inactive objets.
Container components could include outputs requests to write out any variables as the model simulation progresses (Example 8).The output request uses the I/O plugin of the component from which the container inherited its layout (which is always an input component since only input component can define layout).

NextFrAMES run-time engine
The NextFrAMES run-time engine executes the model components in the order as they are defined in the modeling XML (Fig. 3).Processes defined in container component are also executed in the order as they are defined.The first reference implementation of the NextFrAMES run-time engine will be completed in the first quarter of 2009 as part of CCNY contribution to NASA-WaterNet.The reference implementation is designed Introduction

Conclusions References
Tables Figures

Back Close
Full Over time, we would hope to see multiple implementation of the run-time engine to emerge just like there are multiple implementations for web-browsers (Mozilla, Opera, MS Internet Exploder).The different implementation can be optimized for different platforms (ranging from single CPU systems to cluster computers).For instance, an implementation of the NextFrAMES run-time enging for cluster computers could be built on top of ESMF, which already has numerous components needed to manage components with different layouts and couple different domains.While NextFrAMES is not likely to suit every Earth system model needs, but it certainly can provide a strong basis for a wide range of applications.At the minimum, NextFrAMES is an ultimate "Swiss Army Knife" tool for manipulating time varying spatial data products.The ability of interfacing between components at different spatial and temporal frequencies makes NextFrAMES a unique data processing tool that can perform spatial and temporal up and downscaling while performing addition complex calculations.The NextFrAMES modeling XML along with the plugin API has the potential to change the model development paradigm similarly to how HTML changed dramatically the sharing and distributing textual content.

Conclusions
The main difference between NextFrAMES and existing modeling frameworks is that NextFrAMES is intended to provide a high level abstraction of the scientific tasks in model development and hides largely the common functionality that are targeted by other frameworks.
The NextFrAMES modeling XML and plugin infrastructure provides flexible basis to develop wide range of Earth System model components (land surface, ecosystem and hydrological models).Its operational predecessor (FrAMES) demonstrated the feasibility of the new modeling approach and provided flexible development platform.Deficiencies in FrAMES are largely addressed in the NextFrAMES, design ,which will be Introduction

Conclusions References
Tables Figures

Back Close
Full The first implementation of the run-time engine that parses and interprets the modeling XML loads the module and input/output plugins and performs the model simulation is designed for single CPU system, but a parallel version working on shared memory systems is expected by the end of 2009.The parallel and the single CPU implementations of the run-time engine will be able to execute the same modeling XML and corresponding plugins without on single or multi processor machines without modifications.While the current set of input/output plugins are limited to NetCDF data format, additional input/output plugins can be written without affecting the model implementation.
The NextFrAMES run-time engines developed at The City College of New York will be released as open source project through the popular SourceForge archive.The intention is to turn NextFrAMES a freely available model development platform that would attract not only model developers to implement their models within the NextFrAMES infrastructure, but computer scientists to add new features such as data handling capabilities to other file formats beyond NetCDF by developing additional input/output plugins or different implementation of the run-time engine optimized from various platforms.

Conclusions References
Tables Figures

Conclusions References
Tables Figures
Earth System model components Scientific representation of different Earth System model components often express the model structure in a form of block diagram (Fig. 1).The scientific thinking in these block diagrams have two components a) the block diagram layout and b) the processes in the boxes of the block diagram.An ideal modeling framework would allow scientists to focus on these two and alleviate the need to deal with core model functions such as performing model I/O, handling spatial objects and advancing time, or providing visualization.This level of abstraction improves the model documentations and serves as the ultimate means to plug and play compatibility of model components.While the implementation of space and time management in such a modeling framework might be limiting Introduction

-
modeling XML (to describe the overall model structure), module plugin Application Programming Interface (API) for C/C++ (future versions of FrAMES will have bindings to other languages), -I/O plugin API for different file and data services (OpenDAP, WaterML, etc.)run-time engine (currently designed for single CPU system, but we anticipate SMP capabilities next year implemented as part of the last year of our WaterNet effort)

(
Example 4).A process module plugin (that is written in high level programming language, currently only in C) has two functions (initialize, execute).The initialize function is called in the parsing phase of the model execution.The role of the initialize function is to return the list of parameters and input and output variables of the module to NextFrAMES.The process modules can define default value and plausible value range 11 for each requested parameter.Variables can have default value (to overwrite when missing values occur) and unit attributes.The execute function is called in the model calculation phase for each computational object in the container component repeatedly for each time step.The NextFrAMES takes care of converting variables to the requested units and place the parameters and variables into a single user defined systems.SMP capabilities are anticipated by the end of the 2009.
scientific community simultaneously with the publication of the present paper.

Fig. 1 .Fig. 2 .
Fig. 1.An example of a conceptual Hydrological Model following the paths of water in the water cycle.

Example 4 .
Process modules in container components define the source of dynamically loaded plugin file through plugin attribute and the actual module functions within the plugin via the method attributes.Variables and parameters used in the plugin function are mapped to variables in the container component (encapsulating the process module) via alias mechanism.name="akarmi" unit="mˆ3/s" equation="([a] + [b]) * [d]" > <alias name="precip" high resolution land-use/cover information into distribution of cover types (as percentages) at a coarser resolution, or calculate aggregate statistics (mean, minimum, maximum) from fine resolution input to a coarser resolution domain.Container components (Example 3) are the most important elements of the model definition.Containers provide the means to exchange variables from one component downscale Some of the support tools are already under development at The City College of New York supporting and at the University of New Hampshire for both the Global Terrestrial Network for Hydrology (GTN-H, 12 ) effort of the World Meteorological Organization and the end-to-end water resources management tools developed as CCNY/UNH's contribution the the State-of-the Global Water Systems initiative of the Global Water Output request in container components export any variable to the output file specified by the "path" attribute.The output request is carried out using the I/O plugin of the root component of the container component from which it inherited its layout.Introduction Derivative and route operators perform calculations across multiple object relatted to each other through object topology.Derivative operators require 2D or 3D topology to calculate centered, forward or backward difference in x, y or z direction.Route operators sum the values of the specified variable upstream and propagate the accumulated sum downstream.Introduction Input component named "precipitation input" provide the mean to read variables as boundary condition from input source (specified through the "url" attribute).The actual input/output is performed via plugin methods that are dynamically loaded at run time.Figures Output request in container components export any variable to the output file specified by the "path" attribute.The output request is carried out using the I/O plugin of the root component of the container component from which it inherited its layout.