Articles | Volume 18, issue 23
https://doi.org/10.5194/gmd-18-9605-2025
© Author(s) 2025. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/gmd-18-9605-2025
© Author(s) 2025. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
PortUrb: a performance portable, high-order, moist atmospheric large eddy simulation model with variable-friction immersed boundaries
Oak Ridge National Laboratory, 1 Bethel Valley Road, Oak Ridge, TN 37830, USA
Muralikrishnan Gopalakrishnan Meena
Oak Ridge National Laboratory, 1 Bethel Valley Road, Oak Ridge, TN 37830, USA
Kalyan Gottiparthi
Oak Ridge National Laboratory, 1 Bethel Valley Road, Oak Ridge, TN 37830, USA
Nicholson Koukpaizan
Oak Ridge National Laboratory, 1 Bethel Valley Road, Oak Ridge, TN 37830, USA
Stephen Nichols
Oak Ridge National Laboratory, 1 Bethel Valley Road, Oak Ridge, TN 37830, USA
Related authors
Juan M. Restrepo, Matthew Norman, Stuart Slattery, Lawrence Cheung, and Yihan Liu
Wind Energ. Sci. Discuss., https://doi.org/10.5194/wes-2025-208, https://doi.org/10.5194/wes-2025-208, 2025
Manuscript not accepted for further review
Short summary
Short summary
We examine the average power output of a single and a collection of 5 MW wind turbines, mounted on a Tension-Leg Platform (TLP) under the action of fully developed ocean waves. We find that wave motions have a negligible effect on power output.
Daniel Caviedes-Voullième, Mario Morales-Hernández, Matthew R. Norman, and Ilhan Özgen-Xian
Geosci. Model Dev., 16, 977–1008, https://doi.org/10.5194/gmd-16-977-2023, https://doi.org/10.5194/gmd-16-977-2023, 2023
Short summary
Short summary
This paper introduces the SERGHEI framework and a solver for shallow-water problems. Such models, often used for surface flow and flood modelling, are computationally intense. In recent years the trends to increase computational power have changed, requiring models to adapt to new hardware and new software paradigms. SERGHEI addresses these challenges, allowing surface flow simulation to be enabled on the newest and upcoming consumer hardware and supercomputers very efficiently.
Marco A. Giorgetta, William Sawyer, Xavier Lapillonne, Panagiotis Adamidis, Dmitry Alexeev, Valentin Clément, Remo Dietlicher, Jan Frederik Engels, Monika Esch, Henning Franke, Claudia Frauen, Walter M. Hannah, Benjamin R. Hillman, Luis Kornblueh, Philippe Marti, Matthew R. Norman, Robert Pincus, Sebastian Rast, Daniel Reinert, Reiner Schnur, Uwe Schulzweida, and Bjorn Stevens
Geosci. Model Dev., 15, 6985–7016, https://doi.org/10.5194/gmd-15-6985-2022, https://doi.org/10.5194/gmd-15-6985-2022, 2022
Short summary
Short summary
This work presents a first version of the ICON atmosphere model that works not only on CPUs, but also on GPUs. This GPU-enabled ICON version is benchmarked on two GPU machines and a CPU machine. While the weak scaling is very good on CPUs and GPUs, the strong scaling is poor on GPUs. But the high performance of GPU machines allowed for first simulations of a short period of the quasi-biennial oscillation at very high resolution with explicit convection and gravity wave forcing.
Juan M. Restrepo, Matthew Norman, Stuart Slattery, Lawrence Cheung, and Yihan Liu
Wind Energ. Sci. Discuss., https://doi.org/10.5194/wes-2025-208, https://doi.org/10.5194/wes-2025-208, 2025
Manuscript not accepted for further review
Short summary
Short summary
We examine the average power output of a single and a collection of 5 MW wind turbines, mounted on a Tension-Leg Platform (TLP) under the action of fully developed ocean waves. We find that wave motions have a negligible effect on power output.
Daniel Caviedes-Voullième, Mario Morales-Hernández, Matthew R. Norman, and Ilhan Özgen-Xian
Geosci. Model Dev., 16, 977–1008, https://doi.org/10.5194/gmd-16-977-2023, https://doi.org/10.5194/gmd-16-977-2023, 2023
Short summary
Short summary
This paper introduces the SERGHEI framework and a solver for shallow-water problems. Such models, often used for surface flow and flood modelling, are computationally intense. In recent years the trends to increase computational power have changed, requiring models to adapt to new hardware and new software paradigms. SERGHEI addresses these challenges, allowing surface flow simulation to be enabled on the newest and upcoming consumer hardware and supercomputers very efficiently.
Marco A. Giorgetta, William Sawyer, Xavier Lapillonne, Panagiotis Adamidis, Dmitry Alexeev, Valentin Clément, Remo Dietlicher, Jan Frederik Engels, Monika Esch, Henning Franke, Claudia Frauen, Walter M. Hannah, Benjamin R. Hillman, Luis Kornblueh, Philippe Marti, Matthew R. Norman, Robert Pincus, Sebastian Rast, Daniel Reinert, Reiner Schnur, Uwe Schulzweida, and Bjorn Stevens
Geosci. Model Dev., 15, 6985–7016, https://doi.org/10.5194/gmd-15-6985-2022, https://doi.org/10.5194/gmd-15-6985-2022, 2022
Short summary
Short summary
This work presents a first version of the ICON atmosphere model that works not only on CPUs, but also on GPUs. This GPU-enabled ICON version is benchmarked on two GPU machines and a CPU machine. While the weak scaling is very good on CPUs and GPUs, the strong scaling is poor on GPUs. But the high performance of GPU machines allowed for first simulations of a short period of the quasi-biennial oscillation at very high resolution with explicit convection and gravity wave forcing.
Cited articles
Almgren, A., Lattanzi, A., Haque, R., Jha, P., Kosovic, B., Mirocha, J., Perry, B., Quon, E., Sanders, M., Wiersema, D., Willcox, D., Yuan, X., and Zhang, W.: ERF: energy research and forecasting, The Journal of Open Source Software, 8, 5202, https://doi.org/10.21105/joss.05202, 2023. a, b
Bannon, P. R.: On the anelastic approximation for a compressible atmosphere, Journal of the Atmospheric Sciences, 53, 3618–3628, 1996. a
Brown, N., Lepper, A., Weiland, M., Hill, A., Shipway, B., and Maynard, C.: A directive based hybrid met office nerc cloud model, in: Proceedings of the Second Workshop on Accelerator Programming using Directives, 1–8, Association for Computing machinery (ACM), https://doi.org/10.1145/2832105.2832115, 2015. a
Bryan, G. H. and Fritsch, J. M.: A benchmark simulation for moist nonhydrostatic numerical models, Monthly Weather Review, 130, 2917–2928, 2002. a
Castro, I. P., Cheng, H., and Reynolds, R.: Turbulence over urban-type roughness: deductions from wind-tunnel measurements, Boundary-Layer Meteorology, 118, 109–131, 2006. a
Chandrasekaran, S. and Juckeland, G.: OpenACC for programmers: concepts and strategies, Addison-Wesley Professional, https://doi.org/10.5555/3175812, 2017. a
Cook, S.: CUDA programming: a developer's guide to parallel computing with GPUs, Newnes, https://doi.org/10.5555/2430671, 2012. a
Cotton, W. R., Pielke Sr, R. A., Walko, R. L., Liston, G. E., Tremback, C. J., Jiang, H., McAnelly, R. L., Harrington, J. Y., Nicholls, M. E., Carrio, G. G., and McFadden, J. P.: RAMS 2001: Current status and future directions, Meteorology and Atmospheric Physics, 82, 5–29, 2003. a
Droegemeier, K. K.: The Advanced Regional Prediction System (ARPS), storm-scale numerical weather prediction and data assimilation, Meteorol Atmos Phys, 82, 139–170, 2003. a
Droegemeier, K. K., Lazarus, S. M., and Davies-Jones, R.: The influence of helicity on numerically simulated convective storms, Monthly weather review, 121, 2005–2029, 1993. a
Dupont, S. and Brunet, Y.: Edge flow and canopy structure: a large-eddy simulation study, Boundary-Layer Meteorology, 126, 51–71, 2008. a
Durran, D. R.: Improving the anelastic approximation, Journal of the atmospheric sciences, 46, 1453–1461, 1989. a
Fleming, P. A., Gebraad, P. M., Lee, S., van Wingerden, J.-W., Johnson, K., Churchfield, M., Michalakes, J., Spalart, P., and Moriarty, P.: Evaluating techniques for redirecting turbine wakes using SOWFA, Renewable Energy, 70, 211–218, 2014. a
Gardner, D. J., Guerra, J. E., Hamon, F. P., Reynolds, D. R., Ullrich, P. A., and Woodward, C. S.: Implicit–explicit (IMEX) Runge–Kutta methods for non-hydrostatic atmospheric models, Geosci. Model Dev., 11, 1497–1515, https://doi.org/10.5194/gmd-11-1497-2018, 2018. a
Gottlieb, S., Ketcheson, D. I., and Shu, C.-W.: High order strong stability preserving time discretizations, Journal of Scientific Computing, 38, 251–289, 2009. a
Heus, T., van Heerwaarden, C. C., Jonker, H. J. J., Pier Siebesma, A., Axelsen, S., van den Dries, K., Geoffroy, O., Moene, A. F., Pino, D., de Roode, S. R., and Vilà-Guerau de Arellano, J.: Formulation of the Dutch Atmospheric Large-Eddy Simulation (DALES) and overview of its applications, Geosci. Model Dev., 3, 415–444, https://doi.org/10.5194/gmd-3-415-2010, 2010. a
Jasak, H., Jemcov, A., Tukovic, Ž.: OpenFOAM: A C++ library for complex physics simulations, in: International workshop on coupled methods in numerical dynamics, vol. 1000, 1–20, Dubrovnik, Croatia, https://www.croris.hr/crosbi/publikacija/prilog-skup/538019?lang=en (last access: 1 December 2025), 2007. a
Katta, K. K., Nair, R. D., and Kumar, V.: High-order finite volume shallow water model on the cubed-sphere: 1D reconstruction scheme, Applied Mathematics and Computation, 266, 316–327, 2015. a
Khairoutdinov, M. and Kogan, Y.: A new cloud physics parameterization in a large-eddy simulation model of marine stratocumulus, Monthly weather review, 128, 229–243, 2000. a
Lesieur, M. and Metais, O.: New trends in large-eddy simulations of turbulence, Annual review of fluid mechanics, 28, 45–82, 1996. a
Letzel, M. O., Krane, M., and Raasch, S.: High resolution urban large-eddy simulation studies from street canyon to neighbourhood scale, Atmospheric Environment, 42, 8770–8784, 2008. a
Lilly, D. K.: On the application of the eddy viscosity concept in the inertial sub-range of turbulence, NCAR manuscript, 123, University Corporation for Atmospheric Research, UCAR, https://doi.org/10.5065/D67H1GGQ, 1966. a
Lilly, D. K.: The representation of small-scale turbulence in numerical simulation experiments, in: Proc. IBM Sci. Comput. Symp. on Environmental Science, 195–210, IBM Sci. Comput. Symp. on Environmental Science, https://doi.org/10.5065/D62R3PMM, 1967. a
Liu, Y., Miao, S., Zhang, C., Cui, G., and Zhang, Z.: Study on micro-atmospheric environment by coupling large eddy simulation with mesoscale model, Journal of Wind Engineering and Industrial Aerodynamics, 107, 106–117, 2012. a
Lyngaas, I., Norman, M., and Kim, Y.: Sam++: Porting the e3sm-mmf cloud resolving model using a c++ portability library, The International Journal of High Performance Computing Applications, 36, 214–230, 2022. a
Maronga, B., Gryschka, M., Heinze, R., Hoffmann, F., Kanani-Sühring, F., Keck, M., Ketelsen, K., Letzel, M. O., Sühring, M., and Raasch, S.: The Parallelized Large-Eddy Simulation Model (PALM) version 4.0 for atmospheric and oceanic flows: model formulation, recent developments, and future perspectives, Geosci. Model Dev., 8, 2515–2551, https://doi.org/10.5194/gmd-8-2515-2015, 2015. a
Mason, P. J.: Large-eddy simulation: A critical review of the technique, Quarterly Journal of the Royal Meteorological Society, 120, 1–26, 1994. a
Mehta, D., Van Zuijlen, A., Koren, B., Holierhoek, J., and Bijl, H.: Large Eddy Simulation of wind farm aerodynamics: A review, Journal of Wind Engineering and Industrial Aerodynamics, 133, 1–17, 2014. a
Mirocha, J. D., Churchfield, M. J., Muñoz-Esparza, D., Rai, R. K., Feng, Y., Kosović, B., Haupt, S. E., Brown, B., Ennis, B. L., Draxl, C., Sanz Rodrigo, J., Shaw, W. J., Berg, L. K., Moriarty, P. J., Linn, R. R., Kotamarthi, V. R., Balakrishnan, R., Cline, J. W., Robinson, M. C., and Ananthan, S.: Large-eddy simulation sensitivities to variations of configuration and forcing parameters in canonical boundary-layer flows for wind energy applications, Wind Energ. Sci., 3, 589–613, https://doi.org/10.5194/wes-3-589-2018, 2018. a, b, c, d, e, f
Monin, A. S. and Obukhov, A. M.: Basic laws of turbulent mixing in the surface layer of the atmosphere, Contrib. Geophys. Inst. Acad. Sci. USSR, 151, e187, 163–187, 1954. a
Muñoz-Esparza, D., Sauer, J. A., Shin, H. H., Sharman, R., Kosović, B., Meech, S., García-Sánchez, C., Steiner, M., Knievel, J., Pinto, J., and Swerdlin, S.: Inclusion of building-resolving capabilities into the FastEddy® GPU-LES model using an immersed body force method, Journal of Advances in Modeling Earth Systems, 12, e2020MS002141, https://doi.org/10.1029/2020MS002141, 2020. a, b
Muñoz-Esparza, D., Sauer, J. A., Jensen, A. A., Xue, L., and Grabowski, W. W.: The FastEddy® resident-GPU accelerated large-eddy simulation framework: Moist dynamics extension, validation and sensitivities of modeling non-precipitating shallow cumulus clouds, Journal of Advances in Modeling Earth Systems, 14, e2021MS002904, https://doi.org/10.1029/2021MS002904, 2022. a
Norman, M.: portUrb code, Zenodo [code], https://doi.org/10.5281/zenodo.15000787, 2025a. a
Norman, M.: PortUrb: A Performance Portable, High-Order, Moist Atmospheric Large Eddy Simulation Model with Variable-Friction Immersed Boundaries – Dataset, Zenodo [data set], https://doi.org/10.5281/zenodo.15232629, 2025b. a
Norman, M. R.: Targeting atmospheric simulation algorithms for large, distributed-memory, GPU-accelerated computers, in: GPU Solutions to Multi-scale Problems in Science and Engineering, 271–282, Springer, https://doi.org/10.1007/978-3-642-16405-7_17, 2013b. a
Norman, M. R.: A WENO-limited, ADER-DT, finite-volume scheme for efficient, robust, and communication-avoiding multi-dimensional transport, Journal of Computational Physics, 274, 1–18, 2014. a
Norman, M. R.: miniWeather, Tech. rep., Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States), https://doi.org/10.2172/1480685, 2020. a
Norman, M. R., Nair, R. D., and Semazzi, F. H.: A low communication and large time step explicit finite-volume solver for non-hydrostatic atmospheric dynamics, Journal of Computational Physics, 230, 1567–1584, 2011. a
Norman, M. R., Eldred, C., and Gopalakrishnan Meena, M.: Investigating Inherent Numerical Stabilization for the Moist, Compressible, Non-Hydrostatic Euler Equations on Collocated Grids, Journal of Advances in Modeling Earth Systems, 15, e2023MS003732, https://doi.org/10.1029/2023MS003732, 2023b. a, b, c, d, e
O'Neill, W. and Klein, R.: A moist pseudo-incompressible model, Atmospheric research, 142, 133–141, 2014. a
Pedersen, J. G., Gryning, S.-E., and Kelly, M.: On the structure and adjustment of inversion-capped neutral atmospheric boundary-layer flows: Large-eddy simulation study, Boundary-layer meteorology, 153, 43–62, 2014. a
Pressel, K. G., Kaul, C. M., Schneider, T., Tan, Z., and Mishra, S.: Large-eddy simulation in an anelastic framework with closed water and entropy balances, Journal of Advances in Modeling Earth Systems, 7, 1425–1456, 2015. a
Prusa, J. M., Smolarkiewicz, P. K., and Wyszogrodzki, A. A.: EULAG, a computational model for multiscale flows, Computers & Fluids, 37, 1193–1207, 2008. a
Sauer, J. A. and Muñoz-Esparza, D.: The FastEddy® resident-GPU accelerated large-eddy simulation framework: Model formulation, dynamical-core validation and performance benchmarks, Journal of Advances in Modeling Earth Systems, 12, e2020MS002100, https://doi.org/10.1029/2020MS002100, 2020. a, b, c, d, e, f, g, h, i, j, k
Shaw, R. H. and Schumann, U.: Large-eddy simulation of turbulent flow above and within a forest, Boundary-Layer Meteorology, 61, 47–64, 1992. a
Shobaki, G., Kerbow, A., and Mekhanoshin, S.: Optimizing occupancy and ILP on the GPU using a combinatorial approach, in: Proceedings of the 18th ACM/IEEE International Symposium on Code Generation and Optimization, Association for Computing Machinery (ACM), 133–144, doi10.1145/3368826.3377918, 2020. a
Skamarock, W. C. and Klemp, J. B.: The stability of time-split numerical methods for the hydrostatic and the nonhydrostatic elastic equations, Monthly Weather Review, 120, 2109–2127, 1992. a
Skamarock, W. C. and Klemp, J. B.: A time-split nonhydrostatic atmospheric model for weather research and forecasting applications, Journal of computational physics, 227, 3465–3485, 2008. a
Skamarock, W. C., Klemp, J. B., Dudhia, J., Gill, D. O., Liu, Z., Berner, J., Wang, W., Powers, J. G., Duda, M. G., Barker, D. M., and Huang, X. Y.: A description of the advanced research WRF version 4, NCAR tech. note ncar/tn-556+ str, 145, University Corporation for Atmospheric Research (UCAR), https://doi.org/10.5065/1dfh-6p97, 2019. a
Stevens, B., Cotton, W. R., Feingold, G., and Moeng, C.-H.: Large-eddy simulations of strongly precipitating, shallow, stratocumulus-topped boundary layers, Journal of the atmospheric sciences, 55, 3616–3638, 1998. a
Stoll, R., Gibbs, J. A., Salesky, S. T., Anderson, W., and Calaf, M.: Large-eddy simulation of the atmospheric boundary layer, Boundary-Layer Meteorology, 177, 541–581, 2020. a
Tomas, J., Pourquie, M., and Jonker, H.: Stable stratification effects on flow and pollutant dispersion in boundary layers entering a generic urban environment, Boundary-Layer Meteorology, 159, 221–239, 2016. a
Trott, C., Berger-Vergiat, L., Poliakoff, D., Rajamanickam, S., Lebrun-Grandie, D., Madsen, J., Al Awar, N., Gligoric, M., Shipman, G., and Womeldorff, G.: The kokkos ecosystem: Comprehensive performance portability for high performance computing, Computing in Science & Engineering, 23, 10–18, 2021. a, b
van Heerwaarden, C. C., van Stratum, B. J. H., Heus, T., Gibbs, J. A., Fedorovich, E., and Mellado, J. P.: MicroHH 1.0: a computational fluid dynamics code for direct numerical simulation and large-eddy simulation of atmospheric boundary layer flows, Geosci. Model Dev., 10, 3145–3165, https://doi.org/10.5194/gmd-10-3145-2017, 2017. a
Weisman, M. L. and Klemp, J. B.: The dependence of numerically simulated convective storms on vertical wind shear and buoyancy, Monthly Weather Review, 110, 504–520, 1982. a
Weisman, M. L. and Rotunno, R.: The use of vertical wind shear versus helicity in interpreting supercell dynamics, Journal of the atmospheric sciences, 57, 1452–1472, 2000. a
Weller, H., Lock, S.-J., and Wood, N.: Runge–Kutta IMEX schemes for the horizontally explicit/vertically implicit (HEVI) solution of wave equations, Journal of Computational Physics, 252, 365–381, 2013. a
Xie, Z.-T. and Castro, I. P.: Large-eddy simulation for flow and dispersion in urban streets, Atmospheric Environment, 43, 2174–2185, 2009. a
Zarzycki, C. M., Jablonowski, C., Kent, J., Lauritzen, P. H., Nair, R., Reed, K. A., Ullrich, P. A., Hall, D. M., Taylor, M. A., Dazlich, D., Heikes, R., Konor, C., Randall, D., Chen, X., Harris, L., Giorgetta, M., Reinert, D., Kühnlein, C., Walko, R., Lee, V., Qaddouri, A., Tanguay, M., Miura, H., Ohno, T., Yoshida, R., Park, S.-H., Klemp, J. B., and Skamarock, W. C.: DCMIP2016: the splitting supercell test case, Geosci. Model Dev., 12, 879–892, https://doi.org/10.5194/gmd-12-879-2019, 2019. a, b
Zhang, N., Du, Y., and Miao, S.: A microscale model for air pollutant dispersion simulation in urban areas: Presentation of the model and performance over a single building, Advances in Atmospheric Sciences, 33, 184–192, 2016. a
Zhang, W., Almgren, A., Beckner, V., Bell, J., Blaschke, J., Chan, C. Y., Day, M., Friesen, B., Gott, K., Graves, D., Katz, M., Myers, A., Nguyen, T., Nonaka, A., Rosso, M., Williams, S., and Zingale, M.: AMReX: a framework for block-structured adaptive mesh refinement, The Journal of Open Source Software, 4, 1370, https://doi.org/10.21105/joss.01370, 2019. a
Short summary
A new code, portUrb, is described and validated. portUrb is an atmospheric simulation code for turbulent boundary layers including flow through urban areas. The model is coded with an emphasis on robustness, simplicity, readability, portable performance on Graphics Processing Units (GPUs), and rapid prototyping of surrogate models through an ensemble capability where many different configurations can be run simultaneously to explore parameter choices.
A new code, portUrb, is described and validated. portUrb is an atmospheric simulation code for...