Earth system modeling simulates interactions between components of the climate system (e.g., atmosphere, oceans, land surface, sea ice). It plays a critical role in understanding the past and present climate, and in predicting future climate. An increasing number of models have sprung up all over the world, including stand-alone component models and coupled models consisting of multiple component models, such as climate system models (CSMs) and Earth system models (ESMs).

The development of models for Earth system modeling heavily depends on the advancement of computer supports, not only in terms of hardware (such as high-performance computers) but also in terms of software such as compiling setups that include compiler versions and compiler flags. During the continuous evolution of the models, the compiling setups have to be updated frequently for the compatibility of newer high-performance computers with new processors and for better computing performance.

One may think it is easy to update compiling setups, just by installing new compiler version or changing compiler flags. However, it is challenging to update compiling setups for Earth system modeling, because researchers may get significantly different results from the same experiment when using different compiling setups (Liu et al., 2015b). A compiler not only translates the code in a high-level programming language to a low-level language but also tries to improve computational performance of the codes with compiler optimization schemes. Compilers from different families (for example, those in Table 1) and different versions from the same compiler family generally differ in performance optimization schemes as well as the corresponding implementations. On the other hand, different compiler flags of the same compiler version enable and disable different sets of performance optimization schemes. That is why different compiling setups can lead to different results of the same program. The updating of compiling setups therefore will introduce at least two challenges to Earth system modeling. The first challenge concerns reproducibility of simulation results. Due to the chaotic nature of the climate system, more and more studies have shown that trivial round-off errors can trigger significant changes in simulation results of Earth system modeling (Hong et al., 2013; Liu et al., 2015b; Song et al., 2012). Due to the differences of performance optimization schemes among different compiling setups, a change of compiling setups potentially introduces round-off errors. As a result, the results of a simulation obtained with a compiling setup may be irreproducible by another compiling setup.

The second challenge is the reliability of the simulation results. Compilers are large-scale programs with millions of lines of code. It is well understood that with more lines of code there are more potential bugs in the program. Therefore, although there is generally a large amount of software testing before releasing a compiler version, there still can be unknown bugs. Models for Earth system modeling are also large-scale numerical programs with more and more lines of code (Easterbrook and Johns, 2009). There are already ESMs with nearly one million lines of codes (Alexander and Easterbrook, 2015). Therefore, it is possible that some bugs in a compiler version may be triggered by some code segments in a model.

In response to these challenges, several issues about compiling setups should be investigated:

Can different compiling setups achieve the same (bitwise identical) simulation results? If yes, it will be much easier to reproduce previous simulation results.

There are already efforts for the abovementioned issues. It has been demonstrated that with a certain compiler flag, different compiler versions can achieve bitwise identical simulation results for a given model (Liu et al., 2015a). But it is still not known whether compiling setups with the same compiler version but different compiler flags can achieve bitwise identical simulation results. In this paper, we call the compiling setups that can achieve bitwise identical simulation results bitwise identical compiling setups. It is also unknown whether the bitwise identical compiling setups of one model are appropriate for another model. Baker et al. (2015) proposed a new ensemble-based consistency test for the Community Earth System Model (CESM; Hurrell et al., 2013). It can effectively verify whether two compiling setups can achieve consistent simulation results, especially when they do not achieve bitwise identical simulation results. However, we cannot be sure whether a compiling setup is right or wrong. In other words, it cannot help detect compiler bugs. As a result, it is possible that a compiling setup with compiler bugs has been used for the development of a model for a number of years, while a new compiling setup with bug fixes cannot be used for the model development due to the failure in consistency tests.

The results in this paper show that the bitwise identical compiling setup sets of a model can extend to different compiler versions and different compiler flags. They can facilitate the reproduction of original simulation results, assist researchers to determine the compiler flags for model simulations, help researchers build more test cases to detect bugs in models and compilers, and finally improve the reproducibility and reliability of Earth system modeling.

The rest of this paper is organized as follows. Section 2 briefly introduces compiler optimizations. Section 3 shows the bitwise identical compiling setups of three models. Section 4 uses examples to show what can be learned from the comparison of bitwise identical compiling setups between different models. We conclude this paper with discussion in Sect. 5.

Models for Earth system modeling are generally programmed in languages such as Fortran, C, and C++. A number of compilers have been used for Earth system modeling, such as the compiler families listed in Table 1. In the following context, we further introduce the Intel compiler family and GNU Compiler Collection (GCC) with details.

The Intel compiler family, which is developed by the Intel Corporation, is a commercial software product. It has been widely used for Earth system modeling, because most of the high-performance computers for Earth system modeling are equipped with the CPUs manufactured by the Intel Corporation. Table 2 shows the five latest Intel compiler versions (from version 11.1 released in 2009 to version 15.0.1 released in 2014). For each compiler version, there are many compiler optimization options. Table 3 shows several compiler optimization options that may impact the precision of floating-point calculation. They are common to all compiler versions listed in Table 2. For a compiler flag such as “-fp-model”, there may be multiple selections of the values.

GCC is the most widely used free compiler family in the world. Table 4 shows the five latest GCC versions (from version 4.6.4 released in 2013 to version 5.1 released in 2015). For each compiler version, there are also many compiler optimization options. Similar to Table 3, the compiler optimization options in Table 5 may impact the precision of floating-point calculation and are common to all GCC versions listed in Table 4.

In this study, we use three models, namely, Community Atmosphere Model version 5 (CAM5) (Neale et al., 2010), Parallel Ocean Program version 2 (POP2) (Smith et al., 2010), and Flexible Global Ocean- Atmosphere-Land System Model: Grid-point version 2 (FGOALS-g2) (Li et al., 2013a). To obtain bitwise identical compiling setups of a given model, we should first design various compiling setups and then run the model using each of them. In this section, we will briefly introduce the three models, the compiling setups and the bitwise identical compiling setups of each model.

From Tables 8–10 (or Tables 11–13), we can find that, given the same compiler family, bitwise identical compiling setup sets of different models are obviously different. What causes such differences and what can we learn from the differences? To answer these questions, we take the compiling setups of Intel compilers as an example. Based on the results in Tables 8–10, we can generate ideal bitwise identical compiling setup sets (Table 14), following the criterion that if any model achieves bitwise identical results with two different compiling setups, these compiling setups belong to the same ideal bitwise identical compiling setup set. Through comparing Tables 8–10 to 14, we can pose a number of questions; for example,

Regarding all Intel compiler versions, given compiler flag 2 (or 4), why does CAM5 obtain different simulation results when changing compiler optimization level from O0 (or O1) to O2 (or O3)?

Next, we search for answers to the first two questions, namely, what causes such differences and what can we learn from the differences.

This study illustrates that a model can achieve bitwise identical results under different compiling setups. For a given model, there are always a number of bitwise identical compiling setup sets, some of which can be across not only different compiler flags but also different versions of the same compiler family. As a result, the original results with an older compiler version can be exactly reproduced with a newer compiler version. Moreover, the examples in this paper reveal that bitwise identical compiling setup sets can be enlarged through carefully modifying compilation-sensitive code segments, which will facilitate the exact reproduction of original simulation results.

During the development of a model, the model codes increase continuously and need to be tested frequently. The testing can be classified into two categories: scientific testing and technical testing. Scientific testing, which is evaluating the scientific meaning of simulation results, is generally expensive, because it always requires long simulations and requires scientists to evaluate a large amount of results. In contrast, technical testing, which does not depend on the scientific meaning of simulation results, is generally cheap. For example, short simulations (such as several model days) are enough for bitwise identical testing, and bitwise identical testing can be conducted automatically without any burden to scientists (Easterbrook and Johns, 2009). Technical testing therefore should be much more frequent than scientific testing. Since a bitwise identical compiling setup set contains a number of compiling setups that should achieve exactly the same results for a model simulation, it can bring more cases for technical testing. For example, given that a new code version evolves from an old code version with new modifications, the bitwise identical compiling setup sets of each code version can be obtained automatically. If the two code versions do not have the same bitwise identical compiling setup sets, new test cases can be generated for checking why this happens, for example, because of bugs in the codes or compilation-sensitive code segments. If there are compilation-sensitive code segments in the new modifications, we advise researchers to make them insensitive, to make each bitwise identical compiling setup set as big as possible for further development of the model. The first example in Sect. 4.3 reveals that a compilation-sensitive code segment can become insensitive after a slight code modification.

Although the bitwise identical compiling setup sets of different models are generally different, the differences can effectively bring more test cases to detect software bugs in model simulations, especially the bugs of compilers. Although scientists of Earth system modeling generally cannot modify the code of a compiler to fix a bug, they can modify the code of a model to make sure that the model code will not trigger a compiler bug again. For example, based on the differences of bitwise identical compiling setup sets among different models (CAM5, POP2, and FGOALS-g2), we found that a code segment of POP2 triggers a bug of the Intel compiler version 13, and the compiler bug will not be triggered again with a slight modification to the code segment.

There are generally a large number of choices of compiler flags. Researchers may tend to select a compiler flag that can achieve the best computation performance for a model simulation. Our performance evaluation shows that the compiler flag 3 can achieve the best computation performance among the compiler flags in Table 6. According to Tables 8–10, the bitwise identical compiling setup set corresponding to compiler flag 3 is small. It is already known that climate simulation results can be sensitive to round-off errors. To make simulation results most easily reproduced, researchers may be able to use the compiler flag of the best computation performance in a bigger bitwise identical compiling setup set for a model simulation, when the change of compiler flags will not significantly decrease the computation performance. For example, researchers can use the compiler flag -O3 -fp-model strict -fp-speculation=strict -mp1 -no-vec –no-simd for the simulation of the atmosphere models CAM5 and GAMIL2 when the Intel compilers are used, because such a compiler flag does not significantly decrease the computation performance, especially when the number of processes is big (Figs. 4 and 5). Please note that any selection of a compiler flag for a model simulation will not affect the code testing based on bitwise identical compiling setup sets.