Articles | Volume 16, issue 19
https://doi.org/10.5194/gmd-16-5539-2023
© Author(s) 2023. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/gmd-16-5539-2023
© Author(s) 2023. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Development and technical paper
| 
05 Oct 2023
Development and technical paper |
| 05 Oct 2023
| 05 Oct 2023
Comparing the Performance of Julia on CPUs versus GPUs and Julia-MPI versus Fortran-MPI: a case study with MPAS-Ocean (Version 7.1)
Siddhartha Bishnu
CORRESPONDING AUTHOR
Computational Physics and Methods Group, Los Alamos National Laboratory, Los Alamos, NM 87545, USA
Department of Earth, Atmospheric
and Planetary Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
Robert R. Strauss
Computational Physics and Methods Group, Los Alamos National Laboratory, Los Alamos, NM 87545, USA
Center for Nonlinear Studies, Los Alamos National Laboratory, Los Alamos, NM 87545, USA
Mark R. Petersen
Computational Physics and Methods Group, Los Alamos National Laboratory, Los Alamos, NM 87545, USA
Related authors
Mark R. Petersen, Xylar S. Asay-Davis, Alice M. Barthel, Carolyn Branecky Begeman, Siddhartha Bishnu, Steven R. Brus, Philip W. Jones, Hyun-Gyu Kang, Youngsung Kim, Azamat Mametjanov, Brian O’Neill, Kieran K. Ringel, Katherine M. Smith, Sarat Sreepathi, Luke P. Van Roekel, and Maciej Waruszewski
EGUsphere, https://doi.org/10.5194/egusphere-2025-3500, https://doi.org/10.5194/egusphere-2025-3500, 2025
This preprint is open for discussion and under review for Geoscientific Model Development (GMD).
Short summary
Short summary
Ocean models are used to predict currents, temperature, and salinity of the earth’s oceans, much like weather forecasting. As supercomputer hardware changes with evolving technology, models must be updated, and sometimes rewritten. Here we document Omega, a new ocean model that was designed to run on the world’s fastest supercomputers. Testing shows that Omega accurately solves the model equations, and runs efficiently on many different computer architectures, including exascale computers.
Mark R. Petersen, Xylar S. Asay-Davis, Alice M. Barthel, Carolyn Branecky Begeman, Siddhartha Bishnu, Steven R. Brus, Philip W. Jones, Hyun-Gyu Kang, Youngsung Kim, Azamat Mametjanov, Brian O’Neill, Kieran K. Ringel, Katherine M. Smith, Sarat Sreepathi, Luke P. Van Roekel, and Maciej Waruszewski
EGUsphere, https://doi.org/10.5194/egusphere-2025-3500, https://doi.org/10.5194/egusphere-2025-3500, 2025
This preprint is open for discussion and under review for Geoscientific Model Development (GMD).
Short summary
Short summary
Ocean models are used to predict currents, temperature, and salinity of the earth’s oceans, much like weather forecasting. As supercomputer hardware changes with evolving technology, models must be updated, and sometimes rewritten. Here we document Omega, a new ocean model that was designed to run on the world’s fastest supercomputers. Testing shows that Omega accurately solves the model equations, and runs efficiently on many different computer architectures, including exascale computers.
Katherine M. Smith, Alice M. Barthel, LeAnn M. Conlon, Luke P. Van Roekel, Anthony Bartoletti, Jean-Christophe Golaz, Chengzhu Zhang, Carolyn Branecky Begeman, James J. Benedict, Gautam Bisht, Yan Feng, Walter Hannah, Bryce E. Harrop, Nicole Jeffery, Wuyin Lin, Po-Lun Ma, Mathew E. Maltrud, Mark R. Petersen, Balwinder Singh, Qi Tang, Teklu Tesfa, Jonathan D. Wolfe, Shaocheng Xie, Xue Zheng, Karthik Balaguru, Oluwayemi Garuba, Peter Gleckler, Aixue Hu, Jiwoo Lee, Ben Moore-Maley, and Ana C. Ordoñez
Geosci. Model Dev., 18, 1613–1633, https://doi.org/10.5194/gmd-18-1613-2025, https://doi.org/10.5194/gmd-18-1613-2025, 2025
Short summary
Short summary
Version 2.1 of the U.S. Department of Energy's Energy Exascale Earth System Model (E3SM) adds the Fox-Kemper et al. (2011) mixed-layer eddy parameterization, which restratifies the ocean surface layer through an overturning streamfunction. Results include surface layer bias reduction in temperature, salinity, and sea ice extent in the North Atlantic; a small strengthening of the Atlantic meridional overturning circulation; and improvements to many atmospheric climatological variables.
Nairita Pal, Kristin N. Barton, Mark R. Petersen, Steven R. Brus, Darren Engwirda, Brian K. Arbic, Andrew F. Roberts, Joannes J. Westerink, and Damrongsak Wirasaet
Geosci. Model Dev., 16, 1297–1314, https://doi.org/10.5194/gmd-16-1297-2023, https://doi.org/10.5194/gmd-16-1297-2023, 2023
Short summary
Short summary
Understanding tides is essential to accurately predict ocean currents. Over the next several decades coastal processes such as flooding and erosion will be severely impacted due to climate change. Tides affect currents along the coastal regions the most. In this paper we show the results of implementing tides in a global ocean model known as MPAS–Ocean. We also show how Antarctic ice shelf cavities affect global tides. Our work points towards future research with tide–ice interactions.
Milena Veneziani, Wieslaw Maslowski, Younjoo J. Lee, Gennaro D'Angelo, Robert Osinski, Mark R. Petersen, Wilbert Weijer, Anthony P. Craig, John D. Wolfe, Darin Comeau, and Adrian K. Turner
Geosci. Model Dev., 15, 3133–3160, https://doi.org/10.5194/gmd-15-3133-2022, https://doi.org/10.5194/gmd-15-3133-2022, 2022
Short summary
Short summary
We present an Earth system model (ESM) simulation, E3SM-Arctic-OSI, with a refined grid to better resolve the Arctic ocean and sea-ice system and low spatial resolution elsewhere. The configuration satisfactorily represents many aspects of the Arctic system and its interactions with the sub-Arctic, while keeping computational costs at a fraction of those necessary for global high-resolution ESMs. E3SM-Arctic can thus be an efficient tool to study Arctic processes on climate-relevant timescales.
Cited articles
Besard, T., Foket, C., and De Sutter, B.: Effective Extensible Programming:
Unleashing Julia on GPUs,
IEEE T. Parall. Distr., 30, 827–841, https://doi.org/10.1109/TPDS.2018.2872064, 2018. a
Besard, T., Churavy, V., Edelman, A., and De Sutter, B.: Rapid software
prototyping for heterogeneous and distributed platforms,
Adv. Eng. Softw., 132, 29–46, 2019. a
Bezanson, J., Edelman, A., Karpinski, S., and Shah, V. B.: Julia: A fresh
approach to numerical computing, SIAM Rev., 59, 65–98, 2017. a
Bishnu, S.: Time-Stepping Methods for Partial Differential Equations and Ocean
Models, Zenodo,
https://doi.org/10.5281/zenodo.7439539, 2021. a, b, c
Bishnu, S.: Rotating Shallow Water Verification Suite, Zenodo [code],
https://doi.org/10.5281/zenodo.7421135, 2022. a, b, c, d
Bishnu, S., Petersen, M., Quaife, B., and Schoonover, J.: Verification Suite of
Test Cases for the Barotropic Solver of Ocean Models, under review,
https://doi.org/10.22541/essoar.167100170.03833124/v1, 2022. a, b
Bleichrodt, F., Bisseling, R. H., and Dijkstra, H. A.: Accelerating a
barotropic ocean model using a GPU, Ocean Model., 41, 16–21,
https://doi.org/10.1016/j.ocemod.2011.10.001, 2012. a
Byrne, S., Wilcox, L. C., and Churavy, V.: MPI. jl: Julia bindings for the
Message Passing Interface, in: Proceedings of the JuliaCon Conferences,
1, 68, https://doi.org/10.21105/jcon.00068, 2021. a
Caldwell, P. M., Mametjanov, A., Tang, Q., Van Roekel, L. P., Golaz, J. C.,
et al.: The DOE E3SM Coupled Model Version 1: Description and Results at
High Resolution, J. Adv. Model. Earth Sy., 11, 4095–4146,
https://doi.org/10.1029/2019MS001870, 2019. a
Cushman-Roisin, B. and Beckers, J.-M.: Introduction to geophysical fluid
dynamics: physical and numerical aspects, Academic press, ISBN 9780080916781, 2011. a
Dalcín, L., Paz, R., and Storti, M.: MPI for Python,
J. Parallel Distr. Com.,
65, 1108–1115, 2005. a
Dalcín, L., Paz, R., Storti, M., and D’Elía, J.: MPI for Python:
Performance improvements and MPI-2 extensions, J. Parallel Distr. Com., 68,
655–662, 2008. a
Gevorkyan, M. N., Demidova, A. V., Korolkova, A. V., and Kulyabov, D. S.:
Statistically significant performance testing of Julia scientific
programming language, J. Phys Conf. Ser., 1205, 012017,
https://doi.org/10.1088/1742-6596/1205/1/012017, 2019. a
Golaz, J.-C., Caldwell, P. M., Van Roekel, L. P., Petersen, M. R., Tang, Q., Wolfe, J. D., Abeshu, G.,
Anantharaj, V., Asay-Davis, X. S. and Bader, D. C.: The
DOE E3SM Coupled Model Version 1: Overview and Evaluation at
Standard Resolution, J. Adv. Model. Earth Sy., 11, 2089–2129,
https://doi.org/10.1029/2018MS001603, 2019. a
Jiang, J., Lin, P., Wang, J., Liu, H., Chi, X., Hao, H., Wang, Y., Wang, W.,
and Zhang, L.: Porting LASG/IAP Climate System Ocean Model to
Gpus Using OpenAcc, IEEE Access, 7, 154490–154501,
https://doi.org/10.1109/ACCESS.2019.2932443, 2019. a
Klöckner, A., Pinto, N., Lee, Y., Catanzaro, B., Ivanov, P., and
Fasih, A.: PyCUDA and PyOpenCL: A Scripting-Based Approach to GPU Run-Time
Code Generation, Parallel Comput., 38, 157–174,
https://doi.org/10.1016/j.parco.2011.09.001, 2012. a
Klöwer, M., Hatfield, S., Croci, M., Düben, P. D., and Palmer, T. N.: Fluid
Simulations Accelerated With 16 Bits: Approaching 4x Speedup on A64FX by
Squeezing ShallowWaters.jl Into Float16, J. Adv. Model. Earth Sy., 14,
e2021MS002684, https://doi.org/10.1029/2021MS002684, 2022. a
Koldunov, N. V., Aizinger, V., Rakowsky, N., Scholz, P., Sidorenko, D., Danilov, S., and Jung, T.: Scalability and some optimization of the Finite-volumE Sea ice–Ocean Model, Version 2.0 (FESOM2), Geosci. Model Dev., 12, 3991–4012, https://doi.org/10.5194/gmd-12-3991-2019, 2019. a
Lam, S. K., Pitrou, A., and Seibert, S.: Numba: A llvm-based python jit
compiler, in: Proceedings of the Second Workshop on the LLVM Compiler
Infrastructure in HPC, November 2015, 1–6, https://doi.org/10.1145/2833157.2833162, 2015. a
Lin, W.-C. and McIntosh-Smith, S.: Comparing Julia to Performance
Portable Parallel Programming Models for HPC, in: 2021
International Workshop on Performance Modeling, Benchmarking and
Simulation of High Performance Computer Systems (PMBS), IEEE, St. Louis, MO, USA, 15 November 2021, 94–105, https://doi.org/10.1109/PMBS54543.2021.00016, 2021. a
NERSC: Perlmutter architecture specification,
https://docs.nersc.gov/systems/perlmutter/architecture/ (last access: 16 June 2023), 2023. a
Norman, M., Lyngaas, I., Bagusetty, A., and Berrill, M.: Portable C++ Code
that can Look and Feel Like Fortran Code with Yet Another
Kernel Launcher (YAKL),
Int. J. Parall. Prog., 51, 209–230,
https://doi.org/10.1007/s10766-022-00739-0, 2022. a
Perkel, J. M.: Julia: come for the syntax, stay for the speed, Nature, 572,
141–142, https://doi.org/10.1038/d41586-019-02310-3, 2019. a
Petersen, M. R., Jacobsen, D. W., Ringler, T. D., Hecht, M. W., and Maltrud,
M. E.: Evaluation of the Arbitrary Lagrangian–Eulerian
Vertical Coordinate Method in the MPAS-Ocean Model, Ocean Model., 86,
93–113, https://doi.org/10.1016/j.ocemod.2014.12.004, 2015. a
Petersen, M. R., Asay-Davis, X. S., Berres, A. S., Chen, Q., Feige, N.,
Hoffman, M. J., Jacobsen, D. W., Jones, P. W., Maltrud, M. E., Price, S. F.,
Ringler, T. D., Streletz, G. J., Turner, A. K., Van Roekel, L. P., Veneziani,
M., Wolfe, J. D., Wolfram, P. J., and Woodring, J. L.: An Evaluation of
the Ocean and Sea Ice Climate of E3SM Using MPAS and
Interannual CORE-II Forcing, J. Adv. Model. Earth Sy., 11, 1438–1458,
https://doi.org/10.1029/2018MS001373, 2019. a
Petersen, M. R., Bishnu, S., and Strauss, R. R.: MPAS-Ocean Shallow Water
Performance Test Case, Zenodo [code], https://doi.org/10.5281/zenodo.7439134, 2022. a, b, c
Ramadhan, A., Wagner, G. L., Hill, C., Campin, J.-M., Churavy, V., Besard, T.,
Souza, A., Edelman, A., Ferrari, R., and Marshall, J.: Oceananigans.jl: Fast
and friendly geophysical fluid dynamics on GPUs, J. Open Source Softw., 5,
2018, https://doi.org/10.21105/joss.02018, 2020. a
Shchepetkin, A. F. and McWilliams, J. C.: The regional oceanic modeling system
(ROMS): a split-explicit, free-surface, topography-following-coordinate
oceanic model, Ocean Model., 9, 347–404, 2005. a
Srinath, A.: Accelerating Python on GPUs with nvc++ and Cython,
https://developer.nvidia.com/blog/accelerating-python-on-gpus-with-nvc-and-cython/ (last access: 13 December 2022), 2022. a
Strauss, R. R.: Layered Shallow Water Model in Julia, Implementation on CPU,
GPU, and Cluster Hardware, and Performance Tests,
Zenodo [code], https://doi.org/10.5281/zenodo.7493064, 2023. a, b, c
Top 500: Top 500 the list,
https://www.top500.org/lists/top500/2022/06 (last access:
16 June 2023), 2022. a
Trott, C. R., Lebrun-Grandié, D., Arndt, D.,
Ciesko, J., Dang, V., Ellingwood, N., Gayatri, R.,
Harvey, E., Hollman, D. S. and Ibanez, D.: Kokkos 3: Programming Model
Extensions for the Exascale Era, IEEE T. PARALL. Distr., 33, 805–817,
https://doi.org/10.1109/TPDS.2021.3097283, 2022. a
Xu, S., Huang, X., Oey, L.-Y., Xu, F., Fu, H., Zhang, Y., and Yang, G.: POM.gpu-v1.0: a GPU-based Princeton Ocean Model, Geosci. Model Dev., 8, 2815–2827, https://doi.org/10.5194/gmd-8-2815-2015, 2015. a
Zhao, X.-D., Liang, S.-X., Sun, Z.-C., Zhao, X.-Z., Sun, J.-W., and Liu, Z.-B.:
A GPU accelerated finite volume coastal ocean model, J. Hydrodyn.,
29, 679–690, https://doi.org/10.1016/S1001-6058(16)60780-1, 2017. a
Short summary
Here we test Julia, a relatively new programming language, which is designed to be simple to write, but also fast on advanced computer architectures. We found that Julia is both convenient and fast, but there is no free lunch. Our first attempt to develop an ocean model in Julia was relatively easy, but the code was slow. After several months of further development, we created a Julia code that is as fast on supercomputers as a Fortran ocean model.
Here we test Julia, a relatively new programming language, which is designed to be simple to...
