Our experiences in grid computing come from the projects MeteoAG and MeteoAG2 within the national effort AustrianGrid (AGrid), including partners and supercomputer centres from all over Austria . AGrid phase 1 started in 2005 and concentrated on research of basic grid technology and application. Phase 2, started in 2008, continued to build on research of phase 1 and additionally tried to make AGrid self-sustaining. The research aim of this project was not to develop conventional parallel applications that can be executed on individual grid machines, but rather to unleash the power of the grid for single distributed program runs. To simplify this task, all grid sites are required to run a similar Linux operating system. At the height of the project AGrid consisted of nine clusters distributed over five locations in Austria including various smaller sites with ad hoc desktop PC networks. The progress of the project, its challenges and solutions were documented in several technical reports and other publications .

For cloud computing, plenty of providers offer services, e.g. Rackspace or Google Compute Engine. Our cloud computing project RainCloud uses Amazon Web Services (AWS), simply because it is the most well known and widely used. AWS offers different services for computing, different levels of data storage and data transfer, as well as tools for monitoring and planning. The services most interesting for meteorological computing purposes are Amazon Elastic Compute cloud (EC2) for computing and Amazon Simple Storage Service (S3) for data storage. For computing, so-called instances, (i.e. virtual computers) are defined according to their compute power relative to a reference CPU, available memory, storage and network performance.

Figure  shows the basic structure of cloud computing on the right side and AGrid as a grid example on the left side. In both cases an additional layer, so-called middleware, is applied between the compute resources and the end user. The Middleware layer handles all necessary scheduling, transfer of data and set up of cloud nodes. Our middleware is ASKALON , which is described in more detail in Sect. .

In the following sections, we list advantages and disadvantages of grid and cloud concepts, which affected our research most (see Table  for a brief overview). Criteria are extracted from literature, most notably containing a general comparison with all vital issues, , and . The discussed issue of security of sensitive and valuable data did not apply to our research and operational setting. However, for big and advanced operational weather forecasting this might be an issue due to its monetary value. Because the hardware and network is completely out of the end user's control, possible security breaches are harder or even impossible to detect. If security is a concern, detailed discussions can be found in for grid computing, and and for cloud computing.

To make it as simple as possible for a (meteorological) end user to use distributed computing resources, we make use of a so-called middleware system. ASKALON, an existing middleware from the Distributed and Parallel Systems group in Innsbruck, provides integrated environments to support the development and execution of scientific workflows on dynamic grid and cloud environments .

To account for the heterogeneity and the loosely coupled nature of resources from grid and cloud providers, ASKALON has adopted a workflow paradigm based on loosely coupled coordination of atomic activities. Distributed applications are split in reasonably small execution parts, which can be executed in parallel on distributed systems, allowing the runtime system to optimise resource usage, file transfers, load balancing, reliability, scalability and handle failed parts. To overcome problems resulting from unexpected job crashes and network interruptions, ASKALON is able to handle most of the common failures. Jobs and file transfers are resubmitted on failure and jobs might also be rescheduled to a different resource if transfers or jobs failed more than 5 times on a resource (). These features still exist in the cloud version but play a less important role as resources showed to be more reliable in the cloud case.

Figure  shows the design of the ASKALON system. Workflows can be generated in a scientist-friendly Graphical User Interface (GUI) and submitted for execution by a service. This allows for long lasting workflows without the need for the user to be online throughout the whole execution period.

Three main components handle the execution of the workflow: 1.

Scheduler. Activities are mapped to physical (or virtualised) resources for their execution with the end user deciding which pool of resources are used. A wide set of scheduling algorithms is available, e.g. Heterogeneous Earliest Finish Time (HEFT) or Dynamical Critical Path - Clouds (DCP-C) . HEFT, for example, takes as input tasks, a set of resources, the times to execute each task on each resource and the times to communicate results between each job on each pair of resources. Each task is assigned a priority and then distributed onto the resources accordingly. For the best possible scheduling, a training phase is needed to get a function that relates the problem size to the processing time. Advanced techniques in prediction and machine learning are used to achieve this goal ().

MeteoAG started as part of the AGrid computing initiative. Using ASKALON we created a workflow to run a full numerical atmospheric model and visualisation on a grid infrastructure . The model is the non-hydrostatic Regional Atmospheric Modeling System (RAMS; version 6), a fully MPI parallelised Fortran-based code . The National Center for Atmospheric Research (NCAR) graphics library is used for visualisation. Due to all AGrid sites running a similar Linux OS, no special code adaptations to grid computing were needed.

We simulated real cases as well as idealised test cases in the AGrid environment. Most often these were parameter studies testing sensitivities to certain input parameters with many slightly different runs. The investigated area in the realistic simulations covered Europe and a target area over western Austria. Several nested domains are used with a horizontal resolution of the innermost domain of 500 m and 60 vertical levels (approx. 7.5 million grid points). Figure  shows the workflow deployed to the AGrid. Starting with many simulations with a shorter simulation time, it was then decided which runs to extend further. Only runs where heavy precipitation occurs above a certain threshold were chosen. Post-processing done on the compute grid includes extraction of variables and preliminary visualisation, but the main visualisation was done on a local machine.

The workflow characteristics relevant for distributed computing are a few (20–50) model instances but highly CPU intensive as well as lots of interprocess communications. Results of this workflow require a substantial amount of data transfer between the different grid sites and the end user (O(200 Gb)).

Upon investigation of our first runs it was necessary to provide different executables for specific architectures (32 bit, 64 bit, 64 bit Intel) to get optimum speed. We ran into a problem while executing the full model on different architectures. Using the exact same static executable with the same input parameters and set-up led to consistently different results across different clusters . For real case simulations, these errors are negligible compared to errors in the model itself. But for idealised simulations, e.g. investigation of turbulence with an atmosphere initially at rest, where tiny perturbations play a major role, this might lead to serious problems. We were not able to determine the cause of these differences. It seems to be a problem of the complex code of the full model and its interaction with the underlying libraries. While we can only speculate on the exact cause, we strongly advise using a simple and quick test such as simulating an atmosphere at rest or linear orographic precipitation to test for such differences.

MeteoAG2 is the continuation of MeteoAG and also part of AGrid . Based on the experience from the MeteoAG experiments, we hypothesise that it would be much more effective to deploy an application consisting of serial CPU jobs. ASKALON is optimised for submission of single core parts of a workflow, which avoids internal parallelism and communication of activities and allows for the best control over the execution within ASKALON. Thus MeteoAG2 uses a simpler meteorological model, the linear model (LM) of orographic precipitation . The model computes only very simple linear equations of orographic precipitation, is not parallelised, and has short runtime, O(10 s), even with high resolutions (500 m) over large domains. LM is written in Fortran. ASKALON is again used for workflow execution and Matlab routines for visualisation.

With this workflow, rainfall over the Alps was investigated by taking input from the European Centre for Medium-Range Weather Forecasts (ECMWF) model, splitting the Alps into subdomains (see Fig. a) and running the model within each subdomain with variations in the input parameters. The last step combines the results from all subdomains and visualises them. Using grid computing allowed us to run many O(50 000) simulations in a relatively short amount of time O(h). This compares to about 50 typical, albeit a lot more complex runs in current operational meteorological set-ups.

The workflow deployed to the grid (Fig. b) is simple with only two main activities: preparing all the input parameters for all subdomains and then the parallel execution of all runs. One of the drawbacks of MeteoAG2 is the very strict set-up that was necessary due to the state of ASKALON at that time, e.g. no robust if-construct yet, and the direct use of model executables without wrappers. The workflow could not easily be changed to suit different research needs, e.g. change to different input parameters for LM or to using a different model.

In switching to cloud computing, RainCloud uses an extended version of the same simple model of orographic precipitation as MeteoAG2. The main extension to LM is the ability to simulate different layers, while still retaining its fast execution time . The software stack again includes ASKALON, the Fortran-based LM, python scripts and Matplotlib for visualisation.

The inclusion of if-constructs in ASKALON and a different approach to the scripting of activities, (e.g. wrapping the model executables in python scripts and calling these) allows RainCloud to be used in different set-ups. We are now able to run the workflow in three flavours without any changes: idealised, semi-idealised and realistic simulations as well as different settings, operational and research. Figure b depicts the workflow run on cloud computing. Only the first two activities, PrepareLM and LinearModel have to be run, the others are optional. This workflow fits a lot of meteorological applications as it has the following building blocks:

preparation of the simulations (PrepareLM);

All activities are wrapped in Python scripts. As long as the input and output between these activities are named the same, everything within the activity can be changed. We use archives for transfer between the activities, again allowing different files to be packed into these archives.

The operational set-up produces spatially detailed, daily probabilistic precipitation forecasts for the Avalanche Service Tyrol (Lawinenwarndienst Tirol) to help forecast avalanche danger. Figure a shows the schematic of our operational workflow. Starting with data from the ECMWF, we forecast and visualise precipitation probabilities over Tyrol with a spatial resolution of 500 m. Additionally, research type experiments are used to test, explore and run experiments with new developments in LM through parameter studies.

Our workflow set-ups vary substantially in required computation power as well as data size. The operational job is run daily during winter, whereas research types are run in bursts. Data usage within the cloud can be substantial O(500 Gb) with all flavours, but with big differences of data transfer from the cloud back to the local machine. Operational results are small, of the order of O(100 Mb), while research results can amount to O(100 Gb), influencing the overall runtime and costs due to the additional data transfer time.

To define the exact costs for a dedicated server system or the participation in a grid initiative is not trivial, and often even unknown to the provider; we contacted several of them, but due to complicated budgeting methodologies the final costs are not obvious. discussed costs for operating a server environment for data services from a provider perspective, including servers, infrastructure, power requirements and networking. However, the authors did not include the cost of human resources for, e.g., system administration. included human resources and establish a cost model for set-up and maintenance of a data centre. Grids may have different and negotiable levels of access and participation, with varying associated costs to the user. Some initiatives, e.g. PRACE , offer free access to grid resources after a proposal/review process.

Cloud computing on the other hand offers simpler and transparent costs. Pricing varies depending on the provider, capability of a resource and also on the geographical region. Prices (as of November 2014) of AWS on-demand compute instances for Linux OS can be found in Table  and range from USD 0.014 up to ∼5 h-1 (region Ireland). Cheaper instance pricing is available through spot instances where one bids on spare resources. These resources might get cancelled if demand rises, but are a valid option for interruption-tolerant workflows or for developing a workflow.

Figure  shows the difference between spot and on-demand pricing for 25 test runs of our operational workflow (circle and x; right y axis). All runs use 32 cores but a different number of instances, i.e. only one c3.8xlarge (32 cores) instance, but 32 m1.medium (1 core) instances. Runtime only includes the actual workflow, not the spin-up needed to prepare the instances. It usually takes 5–10 s for an instance to become available and another 2–5 min to set up the system and install necessary libraries and software. Spot and on demand only differ in the pricing scheme not in the computational resources themselves. With spot pricing we achieved savings between 65 and 90 %, however with an additional start-up latency of 2–3 min (compared to 5–10 s).

To give an idea, a very simplified cost comparison can be done with the purchasing costs of dedicated hardware, excluding costs for system administration, cooling or power. The operational part of RainCloud runs on 32 cores for approximately 3 h per day for 6 months of the year, i.e. 550 h per year.

A dedicated 32 core server with 64 GB RAM costs around USD 5500 (various brands, excluding tax, Austria, November 2014).

For our operational RainCloud workflow, Fig.  shows the effect of different instance types on the runtime. First, a clear difference between the instance types is evident, with the longest running taking nearly twice as long as the shortest one. Second, even within one instance type, runtime varies by 10–20 percent. Serial execution on a 1 core desktop PC takes about 12 h, i.e. a speedup of ∼18 (a runtime of ∼0.66 h as seen in Fig. ). Based on these experiments our daily operational workflow uses four m3.2xlarge instances.

To put this into relation, showed a speedup for MeteoAG of multiple cores vs. 1 core for a short running test set-up of ∼5, with higher speedups possible for a full complex workflow run. For MeteoAG2, showed a speedup of ∼120 when executing that workflow on several grid machines compared to the execution on a single desktop PC. However, as these are different workflows, no comparison between the type of computing resources can be made from these performance measures.

Different usage scenarios are commonly found in meteorology. For choosing the right type of computing system, several issues need to be taken into account. Only above a certain workflow scale is it worth the effort to move away from a local machine. grids usually have a steep learning curve, clouds offer simple (web) interfaces and local clusters are somewhere in the middle. To make the most out of cloud computing (and to some extent out of grid computing), it is best to have a workflow which can be split into small, independent components.

In an “operational scenario with frequent invocations”, either clouds and grids might be suitable depending on the amount of data transferred and the complexity of the model. Time critical data dissemination of forecast products can be sped up with (data) grids. “Operational scenarios with infrequent invocations” might benefit from using grid or even cloud computing, avoiding the need for a local cluster. Examples are recalculation/reanalysis of seasonal/climate simulations or updating of model output statistics (MOS) equations. One important consideration for operational workflows is the scheduling latency, i.e. the time between submitting a job and its actual execution. and show median latencies of 100 s for Enabling Grids for E-Science in Europe (EGEE) grid, but with frequent outliers upwards to 30 min and more (RainCloud 10–120 s).

For a “research scenario with bursts of high activity with many small tasks”, cloud computing fits perfectly. The costs are fully controllable and only little set-up is required. Examples of such use cases include parameter studies with simple models or computation of MOS. If a lot of data transfer is needed, grid computing is the better alternative. “Research applications with big, long running, data intensive simulations” such as high-resolution complex models are best run on grids or local clusters.