Users of high-performance computing (HPC) have historically enjoyed a steady increase in computing power according to Moore's law, allowing the complexity of HPC applications to increase steadily over time. In recent years, advances in supercomputing power have not come from increasingly fast processors but from increasing the number of individual processors in a system. This has led to HPC applications being redesigned so that they scale well when run on many thousands of processors, and effort will be required to make sure models are efficient on the massively parallel supercomputing architectures of the future. However, even for massively parallel supercomputers, a continuation of the exponential growth of computational power that has been observed in the past appears to be out of reach with current technologies, primarily since power consumption scales linearly with the number of processors such that power demand of future supercomputers will become excessive.

One approach to reducing the energy requirements of the next-generation supercomputer is to use so-called approximate computing or inexact hardware. This is hardware that makes some trade-off between accuracy and/or reproducibility, and speed and/or efficiency. A number of studies have shown the promising nature of such hardware implementations seefor an overview. One of the most promising avenues for inexact computing is the use of a reduction in numerical precision. The precision of a floating-point number can be reduced by decreasing the number of bits used to represent the number. Storing and operating on fewer bits reduces the space in memory required for storage and could speed up computations on suitably designed hardware. Therefore, using a reduced-precision representation for floating-point numbers could be a viable option for reducing the computational cost of complex numerical models, provided the results at reduced precision are acceptable.

The recognized standard for floating-point arithmetic is IEEE 754 floating point . This standard specifies a format using a 32 bit representation for floating-point numbers (often referred to as single precision) and an additional option for a 64 bit double-precision representation. These representations are ubiquitous on modern consumer CPUs. It is possible to trade numerical precision for performance on existing hardware by switching from double to single precision. Using single-precision representations does not necessarily mean computation will be faster; often, floating-point units are designed to perform optimally for 64 bit calculations, but for code that can be vectorized it is possible to process single precision at twice the speed as double precision. A key advantage of using a smaller representation is that more data can fit in memory and, importantly, in cache, which can speed up memory-bound computations, and a smaller data volume is beneficial for parallel applications that communicate between processors via message passing. Significant savings in compute time are observed in many applications when precision is switched from double to single e.g.. A representation using 16 bits has also been standardized by the IEEE, referred to as half precision. It is less common to find support for half precision on consumer hardware, although it is supported on some general purpose graphics processing unit (GPU) hardware. Although consumer CPUs and GPUs typically offer only precision levels supported by IEEE 754, in principal, hardware architectures are not limited to these three precision levels, and custom hardware (e.g. based on field programmable gate arrays – FPGAs) that uses an arbitrary number of bits to represent floating-point numbers could be produced.

Whilst aiming to understand the performance of complex numerical models on new hardware architectures with inexact processing capabilities it is useful to be able to experiment with different precision levels, including those that are not supported natively by currently available hardware. Programmable hardware could be used to study a reduction in precision for an application of interest e.g.. However, this will require a total rewrite of the model to run on such hardware, which will often be a serious programming effort. For most applications, it will be much easier to use software to emulate the behaviour of possible inexact hardware for a model of interest, thus avoiding the cost of re-engineering code for a new hardware platform without knowing what the results might be.

Tools for understanding rounding error in numerical simulations have been developed already. For example, the CADNA library can be used to estimate the contribution of round-off error in numerical calculations and hence determine the likely accuracy of the solution. Along a similar line, the specialized compiler Verificarlo is capable of estimating the number of real bits of precision in a calculation given the errors introduced by round-off errors at various “virtual” precisions. This approach has some problems when investigating rounding errors in weather and climate predictions. For example, for short-term weather simulation, an ensemble of forecasts is typically produced, and each member may have different small-scale features, such as the location of a particular thunderstorm, but all members are valid realizations of the state of the atmosphere. This forecasting methodology may allow us to predict if thunderstorms are likely or not, but not the exact location of an individual storm. If the position of a particular storm is different between a model using high precision and a model using lower precision, results can still be valid for both simulations (as with different ensemble members), but the amount of local precipitation in each model can vary by orders of magnitude. This suggests that identifying the number of bits that are influenced by rounding errors may not be able to tell us what precision we need to make a useful prediction.

A variety of tools also exist for automatically tuning floating-point precision, and producing mixed-precision versions of double-precision programs. Tools such as SAM , Precimonious , CRAFT , and PROMISE all provide automated methods for producing a correct mixed-precision implementation. The automated tuning approach usually requires the specification of an appropriate error threshold used to determine the suitability of a particular automatically generated implementation of a program. This can be straightforward for problems where a numerical method is expected to converge to an exact solution but becomes difficult for models geophysical flows. For these models, there is typically considerable uncertainty in both the formulation of the model and the initial conditions. When we consider chaotic models, which when slightly perturbed will quickly diverge from each other (with the rate of divergence dependent on the weather regime of the initial conditions and varying with time), it becomes difficult to specify such a condition. Two solutions may differ in the most significant bits of the results, yet still represent physically meaningful realizations of the model. In this situation, it becomes necessary to perform explicit simulations and for the evaluation of the results to be performed by a domain scientist who understands the physical behaviour of the model, in order to determine the suitability of a particular reduced-precision implementation.

This paper introduces the rpe (reduced-precision emulator) software , which provides the capability to emulate calculations in arbitrary reduced floating-point precision, so that model developers may understand how reduced floating-point precision affects the result of their simulations. The rpe software allows modellers to study the impact of numerical precision in complex model simulations, while making only small changes to the model code. The software can also be used to identify parts of a program that are problematic for numerical precision and to guide changes to the program to allow a stronger reduction in precision. We refer to this as software co-design and present an example of the co-design of a shallow-water model in Sect. . rpe was developed with a particular focus for the community of weather and climate modelling, but the software could be used with other numerical simulations from different domains to answer the same kind of questions.

Weather and climate models are becoming increasingly important in society. Even though weather and climate models are often run on some of the fastest computing hardware available, this is still not enough computational power to allow them to run at small scales that are believed to be important, for example, for resolving individual convective systems . For weather and climate modelling, it was recently proposed that the use of reduced numerical precision may allow a significant reduction of computing cost . Savings due to increased performance or reduced power consumption could be reinvested into larger computing systems and used to allow simulations at higher spatial resolution. To this end, a number of studies have already been performed on the use of reduced floating-point precision in atmospheric applications that have used ancestors of the rpe tool that is presented in this paper . Results show that numerical precision can be reduced significantly for the applications that have been considered.

Section  presents the rpe library, including an overview on how to use rpe, how rpe works, how floating-point truncation is realized within rpe, how to control numerical precision with rpe, and the basic software architecture. Section  presents results for simulations with reduced numerical precision in a shallow-water model that serves as an example for a possible application of rpe, and Sect.  provides the conclusions.

In this section, we introduce the rpe library and explain how to use the emulator, give details on how the emulator truncates floating-point numbers, and explain how precision will propagate through a simulation. Details of the software architecture and possible pitfalls and limitations of the emulator are also discussed.

In this section, we present a precision analysis for a shallow-water model, a typical medium-sized model set-up that is often studied in the development of atmosphere and ocean models, in a Munk double-gyre configuration. The model under investigation is a standard approach to implement the shallow-water equations on an Arakawa C grid , based on the model used in . The same model has been studied in detail on FPGA hardware . However, this required the model to be completely rewritten to run on the specific hardware. We use the rpe library to make simple tests to determine if precision can be reduced beyond single precision, and how the model can be modified to make it more resilient at lower precisions.

The prognostic parameters in the shallow-water model follow the equations: ∂tu-f+ζ‾yv‾xy+∂xB=ν∇2u+τx,∂tv-f+ζ‾xu‾xy+∂yB=ν∇2v+τy,∂th+∂x(h0+h‾x)u+∂y(h0+h‾y)v=0, where B=gh+12u‾x2+12v‾y2 is the Bernoulli potential, ζ=∂xv-∂yu is relative vorticity, u and v are the zonal and meridional components of the velocity, f is the Coriolis parameter, g is the gravitational acceleration, h0 is the mean fluid depth, h is the surface elevation, and τx and τy represent the wind forcing in zonal and meridional directions. The over-bars indicate averaging in x or y direction on the C grid to map field values between edges and cell centres. The Munk double-gyre test case is a standard model set-up in oceanography, and the specific configuration for this paper is based on model simulations in . The domain size is 3480×3480 km, resolved on a grid with 100×100 points. We use a mean fluid depth h0=500 m and a viscosity of ν=470.23 m2 s-1; the Coriolis parameter describes a beta plane (f=f0+βy with f0=4.46×10-5 s-1 and β=2.0×10-11 m-1 s-1). The wind forcing is 0 in the y direction (τy=0) and τx ranges from -0.4 to 0.2 Pa, varying with latitude in a sinusoidal structure to force the double gyre. A third-order Adams–Bashforth time-stepping scheme with a time step of 25 s is used for time integration.

Since a double-precision experiment acts as a reference, we are aiming to reproduce this reference as closely as possible using reduced precision. A first-order estimation of the effect of reduced precision is achieved by running a single-precision simulation using native 32 bit representation (although a version emulating 23 bit precision has also been run and is of similar quality). Two runs with reduced precision are produced using rpe: one with 15 bit of precision in the floating-point significand and one with 10 bit (the number of significand bits in an IEEE half-precision float).

Each simulation is spun-up for 5 million time steps to reach equilibrium, then integrated for a further 15 million time steps. Figure  shows a snapshot of the model simulations for each of the experiments some time after spin-up. The simulation using single precision (Fig. b) is qualitatively similar to the reference double-precision simulation (Fig. a) in that it shows height variations with similar magnitudes, with no suggestion of numerical stability problems. These snapshots are not expected to be identical between the different precision configurations due to the chaotic nature of the simulations, but similar flow behaviour would be expected. The simulation with emulated 15 bit significands (Fig. c) also shows smooth fields with height features of similar magnitude to the double-precision reference simulation. The surface height field in the simulation with 10 bit significands (Fig. d) has smaller magnitude variations than the reference simulation, and some small-scale noise features can be seen as jagged contours, particularly noticeable in the lower part of the domain. These features suggest the dynamics of the simulation with 10 bit significands are being perturbed by a lack of precision. The problem in the 10 bit experiment is that the increment of the time step is typically much smaller than the magnitude of the prognostic variable. In general, adding a small floating-point number to a large floating-point number can result in a loss of precision. We can avoid this error by maintaining a higher-precision prognostic variable, while computing the expensive right-hand-side tendency with emulated half precision. This scheme is illustrated in Algorithm . In this co-designed 10 bit experimental configuration, all floating-point numbers are half precision except for the prognostic fields and a few parameters used for model initialization (which do not affect the running cost). A snapshot of this experiment is shown in Fig. e. Unlike the original 10 bit experiment, the co-designed 10 bit experiment has a smooth height field showing no indications of numerical stability problems.

Figure  shows the mean height field from the reference double-precision simulation, the 15 bit significand simulation, and the co-designed half-precision simulation. The mean height field is qualitatively similar in all three, suggesting a reduction in floating-point precision is acceptable for this model. We have used the rpe software to determine that the shallow-water model can accept a strong reduction in numerical precision, and thus trading numerical precision against performance appears to be a reasonable approach for this model. In practice, we could now continue to consider implementations of this model on real reduced-precision hardware, such as GPUs supporting 16 bit floating-point arithmetic, or an application-specific hardware implementation.

The shallow-water model provides a good context to discuss the effect of rounding on conservation properties. The surface height integrated over the model domain is conserved in the shallow-water equations. However, in a numerical approximation, we expect that there will be some error introduced due to truncation. For the reference double-precision simulation, the mean surface height of the fluid changes by ∼ 10-17 m over 20 million time steps; in the single-precision simulation, the change is ∼ 10-7 m; and in the co-designed half-precision simulation, it is ∼ 10-3 m. Although the height error at lower precision is increased greatly over the double-precision simulation, the absolute amount of fluid lost is small compared to the domain size. If this is a concern, then a mass-fixing scheme could be introduced to mitigate. Perhaps more interesting is the influence of the rounding scheme used when truncating floating-point significands. If we truncate without rounding (equivalent to always rounding towards 0), a large bias is introduced into the simulation, to the point where it will fail to complete due to numerical errors. If instead we choose to round to the nearest representation (as IEEE but without the special case for numbers halfway between two representations illustrated in Fig. d), we see a height error of ∼ 10-2 m. The difference between the round-to-nearest and the round-to-nearest with tie-to-even rounding schemes is not negligible, highlighting the value of unbiased rounding on computing conserved properties.

A tool for running numerical simulations at emulated reduced floating-point precision has been described and documented. The efficacy of this tool has been demonstrated on a simplified geophysical model, and in particular the value of using rpe to co-design a model for lower precision is demonstrated.

There are a few notable limitations of the emulator which may affect how it can be applied. The lack of support for emulating different exponent sizes may be relevant to certain applications. In principle, it would be possible to extend the software to support emulation of arbitrary exponent sizes below the 11 bit significand of IEEE double precision. This could be done by enforcing a check on the size of each truncated value as it is produced by the truncation procedure. However, these checks will create additional computational overheads.

It is also worth noting that the solution obtained with the emulator is unlikely to be bit reproducible compared to a solution obtained using native hardware support for a given significand size. There are several factors contributing to this, such as the treatment of intrinsics as atomic operations when they may actually be composed of multiple floating-point operations and the ability of compilers to optimize code using the emulator. The emulator operates at the Fortran source level, before compiler optimizations, and therefore code compiled with the emulator is likely to be optimized differently than code not using native reduced precision. For example, the compiler may take advantage of a fused multiply–add (FMA) operation that performs a multiplication and an addition operation in one step with a single rounding at the end, as opposed to rounding an intermediate product to a given number of significand digits before computing the addition and rounding the final result. Such an optimization is unlikely to be available when using the rpe_var type due to the complexity of the overloaded multiplication and addition operators. However, bit reproducibility can also not be guaranteed between simulations that use IEEE double or single precision on different computing hardware or even different compiling flags, so this may not be a primary concern.

Using the emulator for reduced-precision simulations will introduce a significant overhead for all floating-point operations and will result in poorer performance. While the emulator is designed to be used to study a possible precision reduction to allow simulations that run faster on real hardware, it is important to realize that the focus of the software is to be able to verify the output of a model at reduced precision rather than to emulate the actual efficiency gains one might expect from reduced precision. The overhead introduced by the emulator may vary depending on its application. In our experience, the overhead of the library in terms of execution time is at least a factor of 10. Tests on the simple chaotic dynamical system of suggest an execution time penalty of approximately 30 times, compared to a double-precision simulation, and the co-designed shallow-water model detailed in Sect.  experiences an execution time penalty of approximately 70 times compared to the double-precision implementation, both tested using gfortran 4.8 with optimization level 2 to compile both the models and rpe itself.

The rpe software can, in principle, be used as a basis to emulate other hardware set-ups, such as hardware that shows bit flips occasionally. For most hardware set-ups, a simple replacement of the function that is actually realizing the bit truncation will be sufficient. It can also be anticipated that it is possible to change the emulator to mimic the behaviour of specific error patterns of specific reduced-precision hardware e.g..