Articles | Volume 16, issue 3
https://doi.org/10.5194/gmd-16-833-2023
© Author(s) 2023. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/gmd-16-833-2023
© Author(s) 2023. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Parallelized domain decomposition for multi-dimensional Lagrangian random walk mass-transfer particle tracking schemes
Lucas Schauer
CORRESPONDING AUTHOR
Department of Applied Mathematics and Statistics, Colorado School of Mines, Golden, CO 80401, USA
Michael J. Schmidt
Center for Computing Research, Sandia National Laboratories, Albuquerque, NM 87185, USA
Nicholas B. Engdahl
Department of Civil and Environmental Engineering, Washington State University, Pullman, WA 99164, USA
Stephen D. Pankavich
Department of Applied Mathematics and Statistics, Colorado School of Mines, Golden, CO 80401, USA
David A. Benson
Hydrologic Science and Engineering Program, Department of Geology and Geological Engineering, Colorado School of Mines, Golden, CO 80401, USA
Diogo Bolster
Department of Civil and Environmental Engineering and Earth Sciences, University of Notre Dame, Notre Dame IN, 46556, USA
Related authors
No articles found.
Jerome D. Fast, Balwinder Singh, Oscar Diaz-Ibarra, Jeff Johnson, Chandru Dhandapani, Brian Gaudet, Taufiq Hassan, Meng Huang, Jaelyn Litzinger, James Overfelt, Kyle Pressel, Michael Schmidt, Shuaiqi Tang, Adam C. Varble, Hui Wan, Mingxuan Wu, Kai Zhang, and Po-Lun Ma
EGUsphere, https://doi.org/10.5194/egusphere-2026-1538, https://doi.org/10.5194/egusphere-2026-1538, 2026
This preprint is open for discussion and under review for Geoscientific Model Development (GMD).
Short summary
Short summary
We ported a prognostic representation of aerosols to C++ and integrated it into an Earth system model that runs on powerful GPU supercomputers. The code conversion approach keeps the same detailed physics as the Fortran version, was carefully tested, and results show that new code produces aerosol simulations consistent with real‑world data over the central U.S. in spring 2016. Future work will optimize the code for GPUs so to reduce the overall computational time.
Lucas A. McMichael, Michael J. Schmidt, Robert Wood, Peter N. Blossey, and Lekha Patel
Geosci. Model Dev., 17, 7867–7888, https://doi.org/10.5194/gmd-17-7867-2024, https://doi.org/10.5194/gmd-17-7867-2024, 2024
Short summary
Short summary
Marine cloud brightening (MCB) is a climate intervention technique to potentially cool the climate. Climate models used to gauge regional climate impacts associated with MCB often assume large areas of the ocean are uniformly perturbed. However, a more realistic representation of MCB application would require information about how an injected particle plume spreads. This work aims to develop such a plume-spreading model.
Cited articles
Aris, R.: On the dispersion of a solute in a fluid flowing through a tube,
P. Roy. Soc. Lond. A, 235, 67–77, 1956. a
Bear, J.: On the tensor form of dispersion in porous media, J.
Geophys. Res., 66, 1185–1197,
https://doi.org/10.1029/JZ066i004p01185, 1961. a
Bear, J.: Dynamics of Fluids in Porous Media, Dover Publications, ISSN
2212-778X, 1972. a
Beckingsale, D. A., Burmark, J., Hornung, R., Jones, H., Killian, W., Kunen,
A. J., Pearce, O., Robinson, P., Ryujin, B. S., and Scogland, T. R.: RAJA:
Portable Performance for Large-Scale Scientific Applications, in: 2019
IEEE/ACM International Workshop on Performance, Portability and Productivity
in HPC (P3HPC), 71–81, https://doi.org/10.1109/P3HPC49587.2019.00012, 2019. a
Benson, D. A. and Bolster, D.: Arbitrarily Complex Chemical Reactions on
Particles, Water Resour. Res., 52, 9190–9200,
https://doi.org/10.1002/2016WR019368, 2016. a, b
Benson, D. A. and Meerschaert, M. M.: Simulation of chemical reaction via
particle tracking: Diffusion-limited versus thermodynamic rate-limited
regimes, Water Resour. Res., 44, W12201, https://doi.org/10.1029/2008WR007111, 2008. a, b
Benson, D. A., Aquino, T., Bolster, D., Engdahl, N., Henri, C. V., and
Fernàndez-Garcia, D.: A comparison of Eulerian and Lagrangian transport and
non-linear reaction algorithms, Adv. Water Resour., 99, 15–37,
https://doi.org/10.1016/j.advwatres.2016.11.003, 2017. a, b, c
Benson, D. A., Pankavich, S., and Bolster, D.: On the separate treatment of
mixing and spreading by the reactive-particle-tracking algorithm: An example
of accurate upscaling of reactive Poiseuille flow, Adv. Water
Resour., 123, 40–53,
https://doi.org/10.1016/j.advwatres.2018.11.001, 2019. a, b, c
Bentley, J. L.: Multidimensional Binary Search Trees Used for Associative
Searching, Commun. ACM, 18, 509–517, https://doi.org/10.1145/361002.361007, 1975. a
Bolster, D., Paster, A., and Benson, D. A.: A particle number conserving
Lagrangian method for mixing-driven reactive transport, Water Resour. Res., 52, 1518–1527, https://doi.org/10.1002/2015WR018310, 2016. a, b
Crespo, A., Dominguez, J., Barreiro, A., Gómez-Gesteira, M., and Rogers, B.:
GPUs, a New Tool of Acceleration in CFD: Efficiency and Reliability on
Smoothed Particle Hydrodynamics Methods, PLoS ONE, 6, e20685, https://doi.org/10.1371/journal.pone.0020685, 2011. a
Dentz, M., Le Borgne, T., Englert, A., and Bijeljic, B.: Mixing, spreading
and reaction in heterogeneous media: A brief review, J. Contam.
Hydrol., 120–121, 1–17,
https://doi.org/10.1016/j.jconhyd.2010.05.002, 2011. a, b
Ding, D., Benson, D. A., Fernández-Garcia, D., Henri, C. V., Hyndman,
D. W., Phanikumar, M. S., and Bolster, D.: Elimination of the Reaction Rate
“Scale Effect”: Application of the Lagrangian Reactive Particle-Tracking
Method to Simulate Mixing-Limited, Field-Scale Biodegradation at the
Schoolcraft (MI, USA) Site, Water Resour. Res., 53, 10411–10432,
https://doi.org/10.1002/2017WR021103, 2017. a, b
Edwards, H. C., Trott, C. R., and Sunderland, D.: Kokkos: Enabling manycore
performance portability through polymorphic memory access patterns, J. Parall. Distrib. Comput., 74, 3202–3216,
https://doi.org/10.1016/j.jpdc.2014.07.003, 2014. a
Engdahl, N., Schmidt, M., and Benson, D.: Accelerating and Parallelizing
Lagrangian Simulations of Mixing‐Limited Reactive Transport, Water Resour. Res., 55, 3556–3566, https://doi.org/10.1029/2018WR024361, 2019. a, b, c
Engdahl, N. B., Benson, D. A., and Bolster, D.: Lagrangian simulation of mixing
and reactions in complex geochemical systems, Water Resour. Res., 53,
3513–3522, https://doi.org/10.1002/2017WR020362, 2017. a
Gelhar, L. W., Gutjahr, A. L., and Naff, R. L.: Stochastic analysis of
macrodispersion in a stratified aquifer, Water Resour. Res., 15,
1387–1397, https://doi.org/10.1029/WR015i006p01387, 1979. a, b
Gomez-Gesteira, M., Crespo, A., Rogers, B., Dalrymple, R., Dominguez, J., and
Barreiro, A.: SPHysics – development of a free-surface fluid solver – Part
2: Efficiency and test cases, Comput. Geosci., 48, 300–307,
https://doi.org/10.1016/j.cageo.2012.02.028, 2012. a
Graham, S. L., Kessler, P. B., and McKusick, M. K.: Gprof: A Call Graph
Execution Profiler, SIGPLAN Not., 39, 49–57, https://doi.org/10.1145/989393.989401,
2004. a
Herrera, P. A., Massabó, M., and Beckie, R. D.: A meshless method to
simulate solute transport in heterogeneous porous media, Adv. Water
Resour., 32, 413–429,
https://doi.org/10.1016/j.advwatres.2008.12.005, 2009. a
Kennel, M. B.: KDTREE 2: Fortran 95 and C++ Software to Efficiently Search
for Near Neighbors in a Multi-Dimensional Euclidean Space, arXiv Physics,
https://arxiv.org/abs/physics/0408067v2 (last access: 11 August 2022), 2004. a
LaBolle, E. M., Fogg, G. E., and Tompson, A. F. B.: Random-Walk Simulation of
Transport in Heterogeneous Porous Media: Local Mass-Conservation Problem and
Implementation Methods, Water Resour. Res., 32, 583–593,
https://doi.org/10.1029/95WR03528, 1996. a, b
Li, L., Maher, K., Navarre-Sitchler, A., Druhan, J., Meile,
C., Lawrence, C., Moore, J., Perdrial, J., Sullivan, P.,
Thompson, A., Jin, L., Bolton, E. W., Brantley, S. L., Dietrich, W. E., Mayer,
K. U., Steefel, C. I., Valocchi, A., Zachara, J., Kocar, B., Mcintosh,
J., Tutolo, B. M., Kumar, M., Sonnenthal, E., Bao, C., and Beisman,
J.: Expanding the
role of reactive transport models in critical zone processes, Earth-Sci.
Rev., 165, 280–301, 2017. a
Molz, F. and Widdowson, M.: Internal Inconsistencies in Dispersion–Dominated
Models That Incorporate Chemical and Microbial Kinetics, Water Resour. Res.,
24, 615–619, 1988. a
Morvillo, M., Rizzo, C. B., and de Barros, F. P.: A scalable parallel
algorithm for reactive particle tracking, J. Comput. Phys.,
446, 110664, https://doi.org/10.1016/j.jcp.2021.110664, 2021. a
NERSC: Roofline performance model,
https://docs.nersc.gov/tools/performance/roofline/ (last access: 1 January 2023), 2018. a
Norman, M.: YAKL: Yet Another Kernel Library,
https://github.com/mrnorman/YAKL (last access: 11 August 2022), 2022. a
Ofenbeck, G., Steinmann, R., Caparros, V., Spampinato, D. G., and Püschel, M.:
Applying the roofline model, in: 2014 IEEE International Symposium on
Performance Analysis of Systems and Software (ISPASS), 76–85,
https://doi.org/10.1109/ISPASS.2014.6844463, 2014. a
Paster, A., Bolster, D., and Benson, D. A.: Particle tracking and the
diffusion-reaction equation, Water Resour. Res., 49, 1–6,
https://doi.org/10.1029/2012WR012444, 2013. a
Paster, A., Bolster, D., and Benson, D. A.: Connecting the dots:
Semi-analytical and random walk numerical solutions of the
diffusion–reaction equation with stochastic initial conditions, J.
Comput. Phys., 263, 91–112,
https://doi.org/10.1016/j.jcp.2014.01.020, 2014. a
Perzan, Z., Babey, T., Caers, J., Bargar, J. R., and Maher, K.: Local and
Global Sensitivity Analysis of a Reactive Transport Model Simulating
Floodplain Redox Cycling, Water Resour. Res., 57, e2021WR029723, https://doi.org/10.1029/2021WR029723, 2021. a
RAJA Performance Portability Layer: https://github.com/LLNL/RAJA, last access: 11 August 2022. a
Roehl, T., Treibig, J., Hager, G., and Wellein, G.: Overhead Analysis of
Performance Counter Measurements, in: 43rd International Conference on
Parallel Processing Workshops (ICCPW), 176–185,
https://doi.org/10.1109/ICPPW.2014.34, 2014. a
Salamon, P., Fernàndez-Garcia, D., and Gómez-Hernández, J. J.: A
review and numerical assessment of the random walk particle tracking method,
J. Contam. Hydrol., 87, 277–305,
https://doi.org/10.1016/j.jconhyd.2006.05.005, 2006. a, b
Schauer, L.: lschauer95/Parallelized-Mass-Transfer-Domain- Decomposition:
Parallelized Domain Decomposition for Mass Transfer Particle Tracking
Simulations, Zenodo [code], https://doi.org/10.5281/zenodo.6975289, 2022. a
Scheibe, T. D., Schuchardt, K., Agarwal, K., Chase, J., Yang, X., Palmer,
B. J., Tartakovsky, A. M., Elsethagen, T., and Redden, G.: Hybrid multiscale
simulation of a mixing-controlled reaction, Adv. Water Resour., 83,
228–239, https://doi.org/10.1016/j.advwatres.2015.06.006, 2015. a
Schmidt, M. J., Pankavich, S., and Benson, D. A.: A Kernel-based Lagrangian
method for imperfectly-mixed chemical reactions, J. Comput.
Phys., 336, 288–307, https://doi.org/10.1016/j.jcp.2017.02.012,
2017. a
Schmidt, M. J., Pankavich, S. D., and Benson, D. A.: On the accuracy of
simulating mixing by random-walk particle-based mass-transfer algorithms,
Adv. Water Resour., 117, 115–119,
https://doi.org/10.1016/j.advwatres.2018.05.003, 2018. a
Schmidt, M. J., Pankavich, S. D., Navarre-Sitchler, A., and Benson, D. A.: A
Lagrangian Method for Reactive Transport with Solid/Aqueous Chemical Phase
Interaction, J. Comput. Phys., 2, 100021,
https://doi.org/10.1016/j.jcpx.2019.100021, 2019. a, b
Schmidt, M. J., Engdahl, N. B., Pankavich, S. D., and Bolster, D.: A
mass-transfer particle-tracking method for simulating transport with
discontinuous diffusion coefficients, Adv. Water Resour., 140,
103577, https://doi.org/10.1016/j.advwatres.2020.103577,
2020a. a
Schmidt, M. J., Pankavich, S. D., Navarre-Sitchler, A., Engdahl, N. B.,
Bolster, D., and Benson, D. A.: Reactive particle-tracking solutions to a
benchmark problem on heavy metal cycling in lake sediments, J.
Contam. Hydrol., 234, 103642,
https://doi.org/10.1016/j.jconhyd.2020.103642, 2020b. a, b
Schmidt, M. J., Engdahl, N. B., Benson, D. A., and Bolster, D.: Optimal Time
Step Length for Lagrangian Interacting-Particle Simulations of Diffusive
Mixing, Transport Porous Med., 146, 413–433, https://doi.org/10.1007/s11242-021-01734-8, 2022. a
Sole-Mari, G., Fernàndez-Garcia, D., Rodríguez-Escales, P., and Sanchez-Vila,
X.: A KDE-Based Random Walk Method for Modeling Reactive Transport With
Complex Kinetics in Porous Media, Water Resour. Res., 53, 9019–9039,
https://doi.org/10.1002/2017WR021064, 2017. a
Sole-Mari, G., Schmidt, M. J., Pankavich, S. D., and Benson, D. A.: Numerical
Equivalence Between SPH and Probabilistic Mass Transfer Methods for
Lagrangian Simulation of Dispersion, Adv. Water Resour., 126, 108–115,
https://doi.org/10.1016/j.advwatres.2019.02.009, 2019. a, b, c
Sole-Mari, G., Fernandez-Garcia, D., Sanchez-Vila, X., and Bolster, D.:
Lagrangian modeling of mixing-limited reactive transport in porous media;
multirate interaction by exchange with the mean, Water Resour. Res.,
56, e2019WR026993, https://doi.org/10.1029/2019WR026993, 2020. a
Steefel, C. I., Appelo, C. A., Arora, B., Jacques, D., Kalbacher, T., Kolditz,
O., Lagneau, V., Lichtner, P. C., Mayer, K. U., Meeussen, J. C., Molins, S.,
Moulton, D., Shao, H., Šimůnek, J., Spycher, N., Yabusaki, S. B., and
Yeh, G. T.: Reactive transport codes for subsurface environmental
simulation, Comput. Geosci., 19, 445–478,
https://doi.org/10.1007/s10596-014-9443-x, 2015. a
Sun, T., Mitchell, L., Kulkarni, K., Klöckner, A., Ham, D. A., and Kelly,
P. H.: A study of vectorization for matrix-free finite element methods, The
Int. J. High Perform. Comput. Appl., 34,
629–644, https://doi.org/10.1177/1094342020945005, 2020. a
Taylor, G. I.: Dispersion of soluble matter in solvent flowing slowly through a
tube, P. Roy. Soc. Lond. A, 219, 186–203, 1953. a
Tompson, A. and Dougherty, D.: On the Use of Particle Tracking Methods for
Solute Transport in Porous Media, in: Vol. 2 Numerical Methods for Transport
and Hydrologic Processes, edited by: Celia, M., Ferrand, L., Brebbia, C.,
Gray, W., and Pinder, G., vol. 36 of Developments in Water Science, Elsevier,
227–232, https://doi.org/10.1016/S0167-5648(08)70094-7,
1988. a
Tompson, A. F. B., Falgout, R. D., Smith, S. G., Bosl, W. J., and Ashby, S. F.:
Analysis of subsurface contaminant migration and remediation using high
performance computing, Adv. Water Resour., 22, 203–221,
https://doi.org/10.1016/S0309-1708(98)00013-X, 1998. a
Treibig, J., Hager, G., and Wellein, G.: LIKWID: A lightweight
performance-oriented tool suite for x86 multicore environments, in:
Proceedings of PSTI2010, the First International Workshop on Parallel
Software Tools and Tool Infrastructures, San Diego CA, 2010. a
Trott, C. R., Lebrun-Grandié, D., Arndt, D., Ciesko, J., Dang, V., Ellingwood,
N., Gayatri, R., Harvey, E., Hollman, D. S., Ibanez, D., Liber, N., Madsen,
J., Miles, J., Poliakoff, D., Powell, A., Rajamanickam, S., Simberg, M.,
Sunderland, D., Turcksin, B., and Wilke, J.: Kokkos 3: Programming Model
Extensions for the Exascale Era, IEEE T. Parall.
Distr., 33, 805–817, https://doi.org/10.1109/TPDS.2021.3097283, 2022.
a
Valocchi, A. J., Bolster, D., and Werth, C. J.: Mixing-limited reactions in
porous media, Transport Porous Med., 130, 157–182, 2019. a
Williams, S., Waterman, A., and Patterson, D.: Roofline: An Insightful Visual
Performance Model for Multicore Architectures, Commun. ACM, 52, 65–76,
https://doi.org/10.1145/1498765.1498785, 2009. a
Xia, X. and Liang, Q.: A GPU-accelerated smoothed particle hydrodynamics (SPH)
model for the shallow water equations, Environ. Model. Softw.,
75, 28–43, https://doi.org/10.1016/j.envsoft.2015.10.002, 2016. a
Short summary
We develop a multi-dimensional, parallelized domain decomposition strategy for mass-transfer particle tracking methods in two and three dimensions, investigate different procedures for decomposing the domain, and prescribe an optimal tiling based on physical problem parameters and the number of available CPU cores. For an optimally subdivided diffusion problem, the parallelized algorithm achieves nearly perfect linear speedup in comparison with the serial run-up to thousands of cores.
We develop a multi-dimensional, parallelized domain decomposition strategy for mass-transfer...