Articles | Volume 18, issue 14
https://doi.org/10.5194/gmd-18-4535-2025
© Author(s) 2025. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Special issue:
https://doi.org/10.5194/gmd-18-4535-2025
© Author(s) 2025. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
asQ: parallel-in-time finite element simulations using ParaDiag for geoscientific models and beyond
Joshua Hope-Collins
CORRESPONDING AUTHOR
Department of Mathematics, Imperial College London, London SW7 2AZ, UK
Abdalaziz Hamdan
Department of Mathematics, Imperial College London, London SW7 2AZ, UK
Institute for Mathematical Innovation, University of Bath, Bath, BA2 7AY, UK
Werner Bauer
School of Mathematics and Physics, University of Surrey, Guildford, GU2 7XH, UK
Lawrence Mitchell
independent researcher: Edinburgh, UK
Colin Cotter
Department of Mathematics, Imperial College London, London SW7 2AZ, UK
Related authors
No articles found.
Reuben W. Nixon-Hill, Daniel Shapero, Colin J. Cotter, and David A. Ham
Geosci. Model Dev., 17, 5369–5386, https://doi.org/10.5194/gmd-17-5369-2024, https://doi.org/10.5194/gmd-17-5369-2024, 2024
Short summary
Short summary
Scientists often use models to study complex processes, like the movement of ice sheets, and compare them to measurements for estimating quantities that are hard to measure. We highlight an approach that ensures accurate results from point data sources (e.g. height measurements) by evaluating the numerical solution at true point locations. This method improves accuracy, aids communication between scientists, and is well-suited for integration with specialised software that automates processes.
Mariana C. A. Clare, Tim W. B. Leijnse, Robert T. McCall, Ferdinand L. M. Diermanse, Colin J. Cotter, and Matthew D. Piggott
Nat. Hazards Earth Syst. Sci., 22, 2491–2515, https://doi.org/10.5194/nhess-22-2491-2022, https://doi.org/10.5194/nhess-22-2491-2022, 2022
Short summary
Short summary
Assessing uncertainty is computationally expensive because it requires multiple runs of expensive models. We take the novel approach of assessing uncertainty from coastal flooding using a multilevel multifidelity (MLMF) method which combines the efficiency of less accurate models with the accuracy of more expensive models at different resolutions. This significantly reduces the computational cost but maintains accuracy, making previously unfeasible real-world studies possible.
Cited articles
Alnæs, M. S., Logg, A., Ølgaard, K. B., Rognes, M. E., and Wells, G. N.: Unified form language: A domain-specific language for weak formulations of partial differential equations, ACM T. Math. Softw., 40, 1–37, https://doi.org/10.1145/2566630, 2014. a
Amestoy, P., Duff, I. S., Koster, J., and L'Excellent, J.-Y.: A Fully Asynchronous Multifrontal Solver Using Distributed Dynamic Scheduling, SIAM Journal on Matrix Analysis and Applications, 23, 15–41, 2001. a
Amestoy, P., Buttari, A., L'Excellent, J.-Y., and Mary, T.: Performance and Scalability of the Block Low-Rank Multifrontal Factorization on Multicore Architectures, ACM T. Math. Softw., 45, 2:1–2:26, 2019. a
Balay, S., Abhyankar, S., Adams, M. F., Benson, S., Brown, J., Brune, P., Buschelman, K., Constantinescu, E., Dalcin, L., Dener, A., Eijkhout, V., Faibussowitsch, J., Gropp, W. D., Hapla, V., Isaac, T., Jolivet, P., Karpeev, D., Kaushik, D., Knepley, M. G., Kong, F., Kruger, S., May, D. A., McInnes, L. C., Mills, R. T., Mitchell, L., Munson, T., Roman, J. E., Rupp, K., Sanan, P., Sarich, J., Smith, B. F., Zampini, S., Zhang, H., Zhang, H., and Zhang, J.: PETSc/TAO Users Manual, Tech. Rep. ANL-21/39 – Revision 3.21, Argonne National Laboratory, https://doi.org/10.2172/2205494, 2024. a
Bauer, P., Thorpe, A., and Brunet, G.: The quiet revolution of numerical weather prediction, Nature, 525, 47–55, https://doi.org/10.1038/nature14956, 2015. a
Bauer, W., Cotter, C., and Wingate, B.: Higher order phase averaging for highly oscillatory systems, Multiscale Model. Sim., 20, 936–956, 2022. a
Beckett, G., Beech-Brandt, J., Leach, K., Payne, Z., Simpson, A., Smith, L., Turner, A., and Whiting, A.: ARCHER2 Service Description, Zenodo, https://doi.org/10.5281/zenodo.14507040, 2024. a
Bercea, G.-T., McRae, A. T. T., Ham, D. A., Mitchell, L., Rathgeber, F., Nardi, L., Luporini, F., and Kelly, P. H. J.: A structure-exploiting numbering algorithm for finite elements on extruded meshes, and its performance evaluation in Firedrake, Geosci. Model Dev., 9, 3803–3815, https://doi.org/10.5194/gmd-9-3803-2016, 2016. a
Bini, D. A., Latouche, G., and Meini, B.: Numerical Methods for Structured Markov Chains, Oxford University PressOxford, https://doi.org/10.1093/acprof:oso/9780198527688.001.0001, 2007. a
Brezzi, F., Douglas, J., and Marini, L. D.: Two families of mixed finite elements for second order elliptic problems, Numer. Math., 47, 217–235, https://doi.org/10.1007/BF01389710, 1985. a, b
Čaklović, G., Speck, R., and Frank, M.: A parallel implementation of a diagonalization-based parallel-in-time integrator, arXiv [preprint], https://doi.org/10.48550/arXiv.2103.12571, 2021. a, b, c, d
Caldas Steinstraesser, J. G., Peixoto, P. D. S., and Schreiber, M.: Parallel-in-time integration of the shallow water equations on the rotating sphere using Parareal and MGRIT, J. Comput. Phys., 496, 112591, https://doi.org/10.1016/j.jcp.2023.112591, 2024. a
Christlieb, A. J., Macdonald, C. B., and Ong, B. W.: Parallel high-order integrators, SIAM J. Sci. Comput., 32, 818–835, 2010. a
Cotter, C. J.: Compatible finite element methods for geophysical fluid dynamics, Acta Numer., 32, 291–393, 2023. a
Dalcin, L. and Fang, Y.-L. L.: mpi4py: Status update after 12 years of development, Comput. Sci. Eng., 23, 47–54, 2021. a
Dalcin, L. D., Paz, R. R., Kler, P. A., and Cosimo, A.: Parallel distributed computing using Python, Adv. Water Resour., 34, 1124–1139, https://doi.org/10.1016/j.advwatres.2011.04.013, 2011. a
Danieli, F. and Wathen, A. J.: All-at-once solution of linear wave equations, Numer. Linear Algebr., 28, e2386, https://doi.org/10.1002/nla.2386, 2021. a
De Sterck, H., Falgout, R. D., Friedhoff, S., Krzysik, O. A., and MacLachlan, S. P.: Optimizing multigrid reduction-in-time and Parareal coarse-grid operators for linear advection, Numer. Linear Algeb., 28, e2367, https://doi.org/10.1002/nla.2367, 2021. a
De Sterck, H., Falgout, R. D., Krzysik, O. A., and Schroder, J. B.: Efficient Multigrid Reduction-in-Time for Method-of-Lines Discretizations of Linear Advection, J. Sci. Comput., 96, 1, https://doi.org/10.1007/s10915-023-02223-4, 2023b. a, b
De Sterck, H., Falgout, R. D., Krzysik, O. A., and Schroder, J. B.: Parallel-in-time solution of scalar nonlinear conservation laws, arXiv [preprint], https://doi.org/10.48550/arXiv.2401.04936, 2024a. a
De Sterck, H., Falgout, R. D., Krzysik, O. A., and Schroder, J. B.: Parallel-in-time solution of hyperbolic PDE systems via characteristic-variable block preconditioning, arXiv [preprint], https://doi.org/10.48550/arXiv.2407.03873, 2024b. a
De Sterck, H., Friedhoff, S., Krzysik, O. A., and MacLachlan, S. P.: Multigrid reduction-in-time convergence for advection problems: A Fourier analysis perspective, Numer. Linear Algebr., 32, e2593, https://doi.org/10.1002/nla.2593, 2024c. a
Eisenstat, S. C. and Walker, H. F.: Choosing the Forcing Terms in an Inexact Newton Method, SIAM J. Sci. Comput., 17, 16–32, https://doi.org/10.1137/0917003, 1996. a
Embid, P. F. and Majda, A. J.: Low Froude number limiting dynamics for stably stratified flow with small or finite Rossby numbers, Geophys. Astrophys. Fluid Dynam., 87, 1–50, 1998. a
Emmett, M. and Minion, M.: Toward an efficient parallel in time method for partial differential equations, Comm. App. Math. Com. Sci., 7, 105–132, 2012. a
Falgout, R. D. and Schroder, J. B.: A multigrid-in-time algorithm for solving evolution equations in parallel, Tech. rep., Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States), https://doi.org/10.2172/2007260, 2023. a, b
Farrell, P. E., Kirby, R. C., and Marchena-Menéndez, J.: Irksome: Automating Runge–Kutta Time-stepping for Finite Element Methods, ACM Math. Soft., 47, 30:1–30:26, https://doi.org/10.1145/3466168, 2021a. a
Farrell, P. E., Knepley, M. G., Mitchell, L., and Wechsung, F.: PCPATCH: software for the topological construction of multigrid relaxation methods, ACM T. Math. Software, 47, 1–22, 2021b. a
firedrake zenodo: Software used in “asQ: parallel-in-time finite element simulations using ParaDiag for geoscientific models and beyond”, Zenodo [code], https://doi.org/10.5281/zenodo.14205088, 2024. a, b
Gander, M. J.: 50 years of time parallel time integration, in: Multiple Shooting and Time Domain Decomposition Methods: MuS-TDD, Heidelberg, 6–8 May 2013, Springer, 69–113, https://doi.org/10.1007/978-3-319-23321-5_3, 2015. a, b
Gander, M. J. and Halpern, L.: Time parallelization for nonlinear problems based on diagonalization, in: Domain decomposition methods in science and engineering XXIII, Springer, 163–170, https://doi.org/10.1007/978-3-319-52389-7_15, 2017. a, b, c, d
Gander, M. J. and Palitta, D.: A new ParaDiag time-parallel time integration method, SIAM J. Sci. Comput., 46, A697–A718, 2024. a
Gander, M. J., Graham, I. G., and Spence, E. A.: Applying GMRES to the Helmholtz equation with shifted Laplacian preconditioning: what is the largest shift for which wavenumber-independent convergence is guaranteed?, Numer. Math., 131, 567–614, 2015. a
Gander, M. J., Liu, J., Wu, S.-L., Yue, X., and Zhou, T.: ParaDiag: parallel-in-time algorithms based on the diagonalization technique, arXiv [preprint], https://doi.org/10.48550/arXiv.2005.09158, 2021. a, b
Gibson, T. H., McRae, A. T., Cotter, C. J., Mitchell, L., and Ham, D. A.: Compatible Finite Element Methods for Geophysical Flows: Automation and Implementation Using Firedrake, Springer Nature, https://doi.org/10.1007/978-3-030-23957-2, 2019. a
Gibson, T. H., Mitchell, L., Ham, D. A., and Cotter, C. J.: Slate: extending Firedrake's domain-specific abstraction to hybridized solvers for geoscience and beyond, Geosci. Model Dev., 13, 735–761, https://doi.org/10.5194/gmd-13-735-2020, 2020. a
Goddard, A. and Wathen, A.: A note on parallel preconditioning for all-at-once evolutionary PDEs, Verlag der Österreichischen Akademie der Wissenschaften, 135–150, https://doi.org/10.1553/etna_vol51s135, 2019. a
Götschel, S., Minion, M., Ruprecht, D., and Speck, R.: Twelve ways to fool the masses when giving parallel-in-time results, in: Workshops on Parallel-in-Time Integration, Springer, 81–94, https://doi.org/10.1007/978-3-030-75933-9_4, 2020. a
Gray, R. M. et al.: Toeplitz and circulant matrices: A review, Foundations and Trends® in Communications and Information Theory, 2, 155–239, 2006. a
Ham, D. A., Kelly, P. H. J., Mitchell, L., Cotter, C. J., Kirby, R. C., Sagiyama, K., Bouziani, N., Vorderwuelbecke, S., Gregory, T. J., Betteridge, J., Shapero, D. R., Nixon-Hill, R. W., Ward, C. J., Farrell, P. E., Brubeck, P. D., Marsden, I., Gibson, T. H., Homolya, M., Sun, T., McRae, A. T. T., Luporini, F., Gregory, A., Lange, M., Funke, S. W., Rathgeber, F., Bercea, G.-T., and Markall, G. R.: Firedrake User Manual, Imperial College London and University of Oxford and Baylor University and University of Washington, 1st Edn., https://doi.org/10.25561/104839, 2023. a
Hamon, F. P., Schreiber, M., and Minion, M. L.: Parallel-in-time multi-level integration of the shallow-water equations on the rotating sphere, J. Comput. Phys., 407, 109210, https://doi.org/10.1016/j.jcp.2019.109210, 2020. a
Hope-Collins, J., Hamdan, A., Bauer, W., Mitchell, L., and Cotter, C.: Singularity container for “asQ: parallel-in-time finite element simulations using ParaDiag for geoscientific models and beyond”, Zenodo [code], https://doi.org/10.5281/zenodo.14198329, 2024a. a, b
Hope-Collins, J., Hamdan, A., Bauer, W., Mitchell, L., and Cotter, C.: Python scripts for “asQ: parallel-in-time finite element simulations using ParaDiag for geoscientific models and beyond”, Zenodo [code], https://doi.org/10.5281/zenodo.14198294, 2024b. a, b, c, d
Hope-Collins, J., Cotter, C., Hamdan, A., Bauer, W., Mitchell, L., and Ham, D. A.: firedrakeproject/asQ: asQ v0.1, Zenodo [code], https://doi.org/10.5281/zenodo.14592040, 2025. a, b
Horton, G. and Vandewalle, S.: A space-time multigrid method for parabolic partial differential equations, SIAM J. Sci. Comput., 16, 848–864, 1995. a
Kirby, R. C. and MacLachlan, S. P.: Extending Irksome: improvements in automated Runge–Kutta time stepping for finite element methods, arXiv [preprint], https://doi.org/10.48550/arXiv.2403.08084, 2024. a
Kirby, R. C. and Mitchell, L.: Solver Composition Across the PDE/Linear Algebra Barrier, SIAM J. Sci. Comput., 40, C76–C98, https://doi.org/10.1137/17M1133208, 2018. a
Kressner, D., Massei, S., and Zhu, J.: Improved ParaDiag via low-rank updates and interpolation, Numer. Math., 155, 175–209, https://doi.org/10.1007/s00211-023-01372-w, 2023. a
Lange, M., Gorman, G., Weiland, M., Mitchell, L., and Southern, J.: Achieving Efficient Strong Scaling with PETSc Using Hybrid MPI/OpenMP Optimisation, vol. 7905 of Lecture Notes in Computer Science, Springer Berlin Heidelberg, Berlin, Heidelberg, 97–108, ISBN 978-3-642-38749-4, https://doi.org/10.1007/978-3-642-38750-0_8, 2013. a
Legoll, F., Lelievre, T., and Samaey, G.: A micro-macro parareal algorithm: application to singularly perturbed ordinary differential equations, SIAM J. Sci. Comput., 35, A1951–A1986, 2013. a
Liu, J. and Wu, S.-L.: A Fast Block $\alpha$-Circulant Preconditoner for All-at-Once Systems From Wave Equations, SIAM Journal on Matrix Analysis and Applications, 41, 1912–1943, https://doi.org/10.1137/19M1309869, 2020. a
Liu, J., Wang, X.-S., Wu, S.-L., and Zhou, T.: A well-conditioned direct PinT algorithm for first- and second-order evolutionary equations, Adv. Comput. Math., 48, 16, https://doi.org/10.1007/s10444-022-09928-4, 2022. a, b
Maday, Y. and Turinici, G.: A parareal in time procedure for the control of partial differential equations, C. R. Math., 335, 387–392, 2002. a
Majda, A. J. and Embid, P.: Averaging over fast gravity waves for geophysical flows with unbalanced initial data, Theor. Comp. Fluid Dyn., 11, 155–169, 1998. a
May, D. A., Sanan, P., Rupp, K., Knepley, M. G., and Smith, B. F.: Extreme-Scale Multigrid Components within PETSc, in: Proceedings of the Platform for Advanced Scientific Computing Conference, ACM, Lausanne Switzerland, 1–12, ISBN 978-1-4503-4126-4, https://doi.org/10.1145/2929908.2929913, 2016. a
McRae, A. T. T., Bercea, G.-T., Mitchell, L., Ham, D. A., and Cotter, C. J.: Automated Generation and Symbolic Manipulation of Tensor Product Finite Elements, SIAM J. Sci. Comput., 38, S25–S47, https://doi.org/10.1137/15M1021167, 2016. a
Melvin, T., Dubal, M., Wood, N., Staniforth, A., and Zerroukat, M.: An inherently mass-conserving iterative semi-implicit semi-Lagrangian discretization of the non-hydrostatic vertical-slice equations, Q. J. Roy. Meteor. Soc., 136, 799–814, https://doi.org/10.1002/qj.603, 2010. a, b, c
Melvin, T., Benacchio, T., Shipway, B., Wood, N., Thuburn, J., and Cotter, C.: A mixed finite-element, finite-volume, semi-implicit discretization for atmospheric dynamics: Cartesian geometry, Q. J. Roy. Meteor. Soc., 145, 2835–2853, https://doi.org/10.1002/qj.3501, 2019. a
Melvin, T., Shipway, B., Wood, N., Benacchio, T., Bendall, T., Boutle, I., Brown, A., Johnson, C., Kent, J., Pring, S., Smith, C., Zerroukat, M., Cotter, C., and Thuburn, J.: A mixed finite‐element, finite‐volume, semi‐implicit discretisation for atmospheric dynamics: Spherical geometry, Q. J. Roy. Meteor. Soc., 150, 4252–4269, https://doi.org/10.1002/qj.4814, 2024. a
Netterville, N., Fan, K., Kumar, S., and Gilray, T.: A Visual Guide to MPI All-to-all, in: 2022 IEEE 29th International Conference on High Performance Computing, Data and Analytics Workshop (HiPCW), IEEE, 20–27, https://doi.org/10.1109/HiPCW57629.2022.00008, 2022. a
Ruprecht, D.: Wave propagation characteristics of parareal, Computing and Visualization in Science, 19, 1–17, 2018. a
Saad, Y.: A flexible inner-outer preconditioned GMRES algorithm, SIAM J. Sci. Comput., 14, 461–469, 1993. a
Schochet, S.: Fast singular limits of hyperbolic PDEs, J. Differ. Equations, 114, 476–512, 1994. a
Schreiber, M.: Shallow Water Equation Environment for Tests, https://gitlab.inria.fr/sweet/sweet (last access: 14 July 2025), 2018. a
Schreiber, M. and Loft, R.: A parallel time integrator for solving the linearized shallow water equations on the rotating sphere, Numer. Linear Algebr., 26, e2220, https://doi.org/10.1002/nla.2220, 2019. a, b, c
Schreiber, M., Schaeffer, N., and Loft, R.: Exponential integrators with parallel-in-time rational approximations for the shallow-water equations on the rotating sphere, Parallel Comput., 85, 56–65, 2019. a
Skamarock, W. C. and Klemp, J. B.: Efficiency and Accuracy of the Klemp-Wilhelmson Time-Splitting Technique, Mon. Weather Rev., 122, 2623–2630, https://doi.org/10.1175/1520-0493(1994)122<2623:EAAOTK>2.0.CO;2, 1994. a
Speck, R.: Algorithm 997: pySDC - Prototyping Spectral Deferred Corrections, ACM T. Math. Softw., 45, 1–23, https://doi.org/10.1145/3310410, 2019. a
Speck, R. and Ruprecht, D.: Parallel-in-Time Software Survey Results, presented at the 13th Workshop on Parallel-in-time Integration, https://time-x.eu/wp-content/uploads/2024/12/D11-Time_X_SRA.pdf (last access: 21 July 2025), 2024. a
The Met Office: Weather and climate science and services in a changing world, Tech. rep., Met Office, met Office Science and Innovation Strategy document, https://doi.org/10.62998/crmi4887, 2022. a
Vandewalle, S. and Van de Velde, E.: Space-time concurrent multigrid waveform relaxation, Ann. Numer. Math., 1, 335–346, 1994. a
Vargas, D. A., Falgout, R. D., Günther, S., and Schroder, J. B.: Multigrid Reduction in Time for Chaotic Dynamical Systems, SIAM J. Sci. Comput., 45, A2019–A2042, https://doi.org/10.1137/22M1518335, 2023. a
Wathen, A.: Some observations on preconditioning for non-self-adjoint and time-dependent problems, Comput. Math. Appl., 116, 176–180, https://doi.org/10.1016/j.camwa.2021.05.037, 2022. a
Wu, S.-L.: Toward Parallel Coarse Grid Correction for the Parareal Algorithm, SIAM J. Sci. Comput., 40, A1446–A1472, https://doi.org/10.1137/17M1141102, 2018. a
Wu, S.-L. and Zhou, T.: Parallel implementation for the two-stage SDIRK methods via diagonalization, J. Comput. Phys., 428, 110076, https://doi.org/10.1016/j.jcp.2020.110076, 2021. a, b
Wu, S.-L., Zhou, T., and Zhou, Z.: Stability implies robust convergence of a class of preconditioned parallel-in-time iterative algorithms, arXiv [preprint], https://doi.org/10.48550/arXiv.2102.04646, 2021. a
XBraid, LLNL: XBraid: Parallel multigrid in time, https://computing.llnl.gov/projects/parallel-time-integration-multigrid, last access: 14 July 2025. a
Xing, J. Y., Moxey, D., and Cantwell, C. D.: Enhancing the Nektar++ Spectral/Hp Element Framework for Parallel-in-Time Simulations, SSRN, https://doi.org/10.2139/ssrn.5010907, 2024. a
Yamazaki, H., Cotter, C. J., and Wingate, B. A.: Time-parallel integration and phase averaging for the nonlinear shallow-water equations on the sphere, Q. J. Roy. Meteor. Soc., 149, 2504–2513, 2023. a
Executive editor
Parallelization is important for speeding up complex geoscientific
models. In addition to spatial parallelization, several parallel-in-time
(PinT) methods have been developed. This paper introduces the reader to
PinT methods for hyperbolic and geophysical models, and it presents the
asQ library which facilitates the implementation of
diagonalization-based (ParaDiag) methods.
Parallelization is important for speeding up complex geoscientific
models. In addition to...
Short summary
Effectively using modern supercomputers requires massively parallel algorithms. Time-parallel algorithms calculate the system state (e.g. the atmosphere) at multiple times simultaneously and have exciting potential but are tricky to implement and still require development. We have developed software to simplify implementing and testing the ParaDiag algorithm on supercomputers. We show that for some atmospheric problems it can enable faster or more accurate solutions than traditional techniques.
Effectively using modern supercomputers requires massively parallel algorithms. Time-parallel...
Special issue