In this article, we introduce spyro, a software stack to solve wave propagation in heterogeneous domains and perform full waveform inversion (FWI) employing the finite-element framework from Firedrake, a high-level Python package for the automated solution of partial differential equations using the finite-element method. The capability of the software is demonstrated by using a continuous Galerkin approach to perform FWI for seismic velocity model building, considering realistic geophysics examples. A time domain FWI approach that uses meshes composed of variably sized triangular elements to discretize the domain is detailed. To resolve both the forward and adjoint-state equations and to calculate a mesh-independent gradient associated with the FWI process, a fully explicit, variable higher-order (up to degree

The construction of models consistent with observations of Earth's physical properties can be posed mathematically as solving an inverse problem referred to as full waveform inversion (FWI)

The basic method of FWI requires several computationally and memory expensive components that need to be executed iteratively potentially dozens of times to arrive at an optimized model

Considering the computational cost of solving the wave equation is important to efficiently performing FWI, finite-difference methods are often used to model wave propagation. Finite-difference methods are well studied in the context of seismic application in part because they can be highly optimized for computational performance, especially so with the help of recent packages such as Devito

Despite the advantage unstructured meshes appear to offer to FWI, there are several major difficulties associated with using them that we attempt to address in this work: (1) the computational burden associated with solving a sparse system of equations arising from the discretization with finite elements, (2) the generation and distribution of variable-resolution unstructured meshes, and (3) code complexity and optimization associated with programming finite-element methods themselves. Unlike in the case of SEMs, in which the domain is discretized using tensor-based hexahedral elements that result in diagonal mass matrices (e.g., mass lumped) and can be efficiently time marched

To address the first issue, we point out that certain triangular finite-element spaces do admit diagonal approximations to mass matrices.
These spaces contain the standard set of polynomials of some degree

A second major difficulty is the generation and design of a variable-resolution triangular mesh. This can be a potentially laborious mesh generation pre-processing step and can strongly limit the applicability of the method, especially in 3D (e.g.,

Third, the high complexity of implementing efficient unstructured FEMs frequently discourages domain practitioners.
Compared to finite-difference methods, FEMs require additional levels of coding complexity associated with mesh data structures, numerical integration, function spaces, matrix assembly, and sophisticated code optimizations for looping over unstructured mesh connectivity

The Firedrake project

The aim of this paper is to address the issues associated with the application of triangular, unstructured FEMs to perform FWI with the higher-order mass-lumped elements of

The article is organized as follows: first we introduce the FWI algorithm and discuss the continuous formulation. Thereafter, we focus on the discretization of the governing equations in both space and time. Following this, we discuss our Firedrake implementation. Then we study the error associated with discretizing the domain with variable-resolution triangular meshes. Lastly, we demonstrate computational results in both 2D and 3D, discuss, and conclude the work.

Figure

In a typical field setup in an offshore/marine environment, a ship tows a cable potentially several kilometers long with hundreds of microphones (Fig.

A simplified illustration of a marine seismic survey with relevant components annotated. Courtesy from João Baptista Dias Moreira.

FWI can either be posed in the time domain or frequency domain

One key challenge associated with FWI and inverse problems in general is that they require an adequate starting velocity model to converge toward the global minimum of the misfit. In other words, the initial model should be able to predict the travel time of any arrival involved in the inversion to within half a period of the lowest inverted frequency when a classical least-squares misfit function based on the data difference is used; otherwise, the FWI will converge to a local minimum (e.g.,

In this work, the acoustic wave equation in its second-order form is considered in either a 2D or 3D physical domain

When simulated waves reach the extent of the domain, they create reflections generating signals that are deleterious for FWI applications since field data do not contain these signals. Thus, in this work an absorbing boundary layer referred to as a perfectly matched layer (PML) is included as a small domain extension

The coupled system of equations for the modified acoustic wave equation with the PML is given by the residual operators

In 2D, the modified acoustic wave formulation is simplified since

All sources

In this section, the optimization components of the FWI process are detailed. Experimental data are generated by exciting a physical domain

For a given velocity model

This Lagrangian is dependent on the forward solution

In addition to these volume equations, we can deduce the boundary and initial/final conditions for their variables. One can verify that a homogeneous final condition (on

We have discretized the modified acoustic equation (Eqs.

CG methods actually provide a family of methods, parameterized over the choice of approximating spaces rather than a single method. Frequently, the choice of approximating spaces only affects the overall accuracy – by choosing standard

Regardless of the particulars, we denote the finite-element function space used within our CG method as

For linear triangular elements, mass lumping can be accomplished using the standard Lagrange basis functions and vertex-based Newton–Cotes integration rule. However, for higher-degree (

The aforementioned concept of using higher-order bubble functions to achieve these elements is illustrated and compared with standard Lagrange elements in both 2D and 3D in Figs.

Some two-dimensional Lagrange and ML elements.

Some three-dimensional Lagrange and ML elements.

In order to efficiently discretize the domain, a triangular mesh of conforming elements interchangeably referred to as a mesh has to be generated. The major benefit of this approach is that mesh elements range in size according to several aspects elaborated below (e.g., Fig.

The design of a so-called optimal mesh in a way that maximizes accuracy while minimizing computational cost through mesh size variation represents a challenging task. One crucial aspect is the numerical stability condition, which puts constraints on meshing because the time step is affected by the smallest cell via the Courant–Friedrichs–Lewy (CFL) condition (e.g.,

The Marmousi2

In this work, variable-resolution element sizes are based on the acoustic wavelength, the CFL condition, and a mesh gradation rate. Altogether the design of resolution becomes proportional to the wavelength of the acoustic wave, hence the phrase waveform adapted. The assumption is made that all triangles will be nearly equilateral, which is necessary for accurate simulation with FEMs. The mesh can be the result of any external mesh generator; in this work we use a domain-specific mesh generator tool called SeismicMesh

The length of the element's edges

The selection of

A second-order accurate fully explicit central finite-difference scheme was used to discretize all time derivative terms. While higher-order time-stepping schemes such as a Dablain scheme

For a given variable

In a practical application, it remains important to be able to determine a numerically stable time step for this discretization, and this depends on element degree

In practice, a time step

The numerical implementation of the adjoint problem and the gradient computation are detailed in this section. From the adjoint equations presented in their strong form (Eqs.

In addition to the adjoint, the gradient is computed by discretizing Eq. (

In order to derive the discrete adjoint and gradient, the time integrals appearing in the continuous formulation (i.e., in the cost functional

Also, we stress here that, in order to obtain the gradient

Numerical simulations often require several thousand time steps integrating over several seconds with the aforementioned numerical approaches to compute the discrete gradient. As a result, there are significant memory requirements for storing the forward-state solution that is necessary to calculate

In this section, important components of our implementation are explained. The Firedrake package is used to implement the numerical developments. The code

An additional layer of implementation is necessary for applications in seismic problems as there are operations to execute FWI that fall outside of the capabilities within the Firedrake package. It is important to mention that Firedrake has components that allow us to compute the gradient with automatic differentiation (AD) (e.g., dolfin-adjoint,

The Rapid Optimization Library (ROL;

As mentioned earlier in this work, we exclusively rely on the second-order optimization method L-BFGS

Given the gradient subsampling (see Sect.

Five triangular elements for spatial polynomial degrees

The quadrature rules are key to defining the finite-element basis, so we began by implementing these within FIAT. Then, to define the finite elements themselves, we must first construct the function space. This is done using two particular FIAT features described in

Certain standard boilerplate is required to expose a new FIAT element to the rest of Firedrake. First, the element, along with certain metadata, must be announced within the Unified Form Language. Then, it must also be wrapped into FInAT

To probe the computational domain, functionality is required to both record the solution at a set of points (i.e., receivers) and to inject the domain with a time-varying wavelet (i.e., sources; Fig.

Source injection is the adjoint operator of interpolating the solution to the receivers. To execute both, a Dirac mass is integrated against the finite-element basis functions in the form of weights equal to the basis functions evaluated at the source (receiver) position inside the element that contains the source (receiver). This point force source is of the form

We start with an initial distribution of

The discussed inversion process is shown in Algorithm 1. In order to ensure a sufficient decrease of the objective functional at each inversion iteration

The spatial and temporal discretizations detailed in Sect.

The functionality of the forward-state, adjoint-state, and gradient Python codes.

We point out that all spatial discretizations are performed using matrix-free approaches, which are available in the Firedrake computing environment, and this reduces runtime memory requirements (e.g.,

A two-level parallelism strategy is implemented over both the sources and spatial domain decomposition. In space, domain decomposition parallelism is handled by the Firedrake library, which automatically handles setting up halo/ghost zones around each subdomain and performing the necessary communication at each time step via the Message Passing Interface (MPI). In addition, Firedrake also provides options to configure the depth of the ghost layer for performance as needed. In this work, no ghost layers are added to the subdomains, and instead the solution is shared only at the boundary nodes of each subdomain. At the source level, parallelism is trivial and handled by splitting the MPI communicator into groups of processes at initialization and assigning each group to simulate one source. Due to the usage of Firedrake, no additional code is required for parallelism as compared to the sequential version of the code.

We do note a significant benefit from using both shot-level and domain decomposition parallelism simultaneously, especially in 3D, which is later detailed in Sect.

Mesh files are read in from disk sequentially and then distributed in parallel if necessary; this functionality is handled latently by Firedrake. External seismic velocity models are read in from disk from an H5 file format at execution time. Gridded velocity data are bi-linearly interpolated onto the nodal DoFs of the elements of the mesh at runtime. In this way, seismic velocities can vary inside the element in the event higher-order elements are used. Gridded seismic velocity files can be prepared using SeismicMesh

The accurate computation of the discrete gradient is crucial for the robustness of Algorithm 1. Through a numerical experiment, we demonstrate that the gradients computed through the optimize-then-discretize approach (see Sect.

The considered 2D test problem to verify the numerical gradients was a physical domain

In 3D a similar problem to the 2D case was considered within a physical domain

For both 2D and 3D cases, an initial velocity model with a uniform velocity of

Comparison of directional derivatives for 2D and 3D cases between the finite-difference approximation (fd) and our discrete gradient (co).

We point out that relatively good agreement was found between the two derivatives in which the wave field was resolved with at least

Problem configuration for verification of the discrete gradient in

To effectively apply higher-order mass-lumped methods with unstructured meshes, it is important to understand the required mesh resolution for a given desired accuracy

The implementation of the forward-state wave propagator in 2D and 3D was first verified in order to reliably intercompare solutions between elements. An equivalence was demonstrated between a converged numerical result computed on a highly refined mesh in 2D and 3D and compared with their analytical solutions, respectively. Following that, the assumption was made that equivalence holds for all our subsequent tests, implying that all the reference waveforms are converged numerical solutions, given sufficiently fine mesh resolution and sufficiently small numerical time steps.

The method of manufactured solutions (MMS) was used to verify the implementation in accordance with a manufactured analytical solution. The manufactured 2D analytical solution was chosen as

Our experiments demonstrated good agreement between the analytical and modeled solutions with a relative error for the 2D reference homogeneous case of

A 2D wave propagation experiment in a domain with a homogeneous velocity field was configured to quantify the accuracy of the forward-state solution with ML elements (Fig.

Error was calculated based on the simulated pressure recorded at the receiver locations with

The space of

The experimental configuration to calculate the grid-point-per-wavelength

Panels

Results from the grid sweep for both the homogeneous and heterogeneous experiment to identify efficient values for

Overall, the homogeneous grid sweep results demonstrate that elements with spatial order

Applying the results from the homogeneous grid sweep, the values for

In addition to generating a mesh that meets the requirements of the technique used to numerically discretize the PDE, the mesh must also account for local variations in the seismic velocity, which can have significant effects on the simulation of acoustic waves. In the case of simulation with a heterogeneous velocity model,

As a result, in this experiment we added an additional percent to the parameter values of

For triangular meshes, the

In a similar manner to the experiment with the homogeneous velocity model, a 2D experiment with a heterogeneous velocity model was performed for the BP2004

In the mesh generation process, a mesh gradation rate of

The reference problem configuration for the BP2004 seismic velocity model. Panel

As shown by Fig.

Wave propagation errors can be the result of dispersion and also show how well the mesh represents the local seismic wave speed variations. In our mesh design, exact fault locations were not resolved with edge-orientated elements (e.g.,

For ML3tri, ML4tri, and ML5tri elements, peak

Panel

A similar experiment to that described Sect.

The 3D experimental configuration to calculate the grid-point-per-wavelength

The results are shown in Fig.

The 3D grid sweep, similar to Fig.

Simulations were executed on a cluster called Mintrop at the University of São Paulo. Experiments used four Intel-based computer nodes. Each Intel node was a dual-socket Intel Xeon Gold

Strong scaling curves for solving the acoustic wave equation with a PML in spyro for 2D

The parallel efficiency of our forward propagator was assessed in Intel-based CPUs (see Fig.

The 3D domain was

Overall, nearly ideal strong scaling was observed in both 2D and 3D cases for most of the elements tested up to

To investigate FWI (Algorithm 1) with variable unstructured meshes, several 2D inversions were performed using the Marmousi2 model

The Marmousi2 model setup described in Sect.

FWIs commenced from an initial

Each inversion used an acquisition geometry setup of

The FWI setup described in Algorithm 1 was run for a maximum of

Except for the ML1tri experiment, all meshes for the initial velocity model were generated using the

The number of degrees of freedom (DoFs) for each experiment, the cells per wavelength

The simulations were performed using one shot per core using the shot-level ensemble parallelism described in Sect.

The number of DoFs varied by approximately a factor of 2 over the range of ML elements tested. As expected, the ML1tri produced the largest problem size with

The final inverted models are shown in Fig.

The total runtime memory and wall-clock varied substantially (Fig.

The final result for each FWI using different ML elements. The total number of iterations (including both iterations that reduced the cost functional and the ones that did not) are indicated in each figure along with the final

Comparing the performance of FWIs computed with different ML elements. Panel

As a demonstration of all the previous developments, the FWI implementation was applied to invert a section of the Overthrust3D

For the inversion,

Both the guess and true velocity models were discretized with ML3tet elements. Each model featured elements adapted in size to the true and guess model's local seismic velocity given a

Similar to the 2D FWI experiment, the 3D FWI ran for a maximum of

Simulations were performed using the two-level parallelism strategy with two AMD nodes. Each AMD-based node had an AMD EPYC 7601 machine with

The Overthrust3D setup described in Sect.

The final inversion results along several cross-sectional slices along the

Even with the use of mass-lumping elements and variable mesh resolution, 3D FWI remains computationally challenging on a relatively small-scale cluster with

A comparison of cross-sectional slices along the

Same as Fig.

We have discussed a methodology for imaging regional seismic velocity in two- and three-dimensional, arbitrarily heterogeneous, semi-infinite domains in a process commonly referred to as full waveform inversion (FWI). The FWI process involves solving a PDE-constrained optimization problem to minimize the misfit between the collected data and the computed response of the forward equations starting from some initial distribution of seismic velocity. To solve this problem, a continuous Galerkin (CG) finite-element method (FEM) approach was developed using unstructured triangular (i.e., in 2D and tetrahedral in 3D) meshes with elements adapted in size to local seismic velocity. The forward, the adjoint-state wave fields, and the gradient are computed on the same unstructured triangular mesh.
The FWI was implemented using the Firedrake package

Five triangular 2D elements and three tetrahedral 3D elements that were originally detailed in

The work presented in this article presents several new directions for FWI with triangular FEM. In the course of the FWI process, the physical model incrementally evolves, and to aid convergence towards the global minimum of the cost function, a multi-scale reconstruction is often used by increasing step by step the frequency of the simulated phenomena. In the case of multi-scale FWI, an automated meshing process in the FWI loop is then crucial to deal with the variations of the physical parameters and the increase of the frequency component of the waves simulated. Waveform-adapted meshes could be used for each frequency of interest so as to obtain an accurate solution while using the coarsest mesh possible.

The current package enables developers to implement forward wave solvers that each make different physical assumptions (variable density acoustic, elastic, visco-elastic, etc.) and discretize them using Firedrake within the current API. In this extensible environment, automatic differentiation (AD) (e.g., dolfin-adjoint,

In this section, we include a few steps on the continuous adjoint derivation, given in the body of the text in Eqs. (

The expression of the matrices used in the forward discrete problem is

All code used in this repository is free and open source, and all data sets used in the demonstrations are publicly available. The spyro source repository is available from

KJR, AO, and LF implemented the code, developed the numerical formulation, performed several results for the paper, and together wrote the paper. RCK helped implement some aspects of the finite elements used and wrote and improved the paper. BSC and RSG provided the research environment and intellectual discussion necessary for the software's development and eventual realization of this paper.

The contact author has declared that none of the authors has any competing interests.

Publisher’s note: Copernicus Publications remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This research was carried out in association with the ongoing R&D project registered as ANP 20714-2, “Software technologies for modeling and inversion, with applications in seismic imaging” (University of São Paulo/Shell Brasil/ANP) – “Desenvolvimento de técnicas numéricas e software para problemas de inversão com aplicações em processamento sísmico”, sponsored by Shell Brasil under the ANP R&D levy as “Compromisso de Investimentos com Pesquisa e Desenvolvimento”.

The fourth author (RCK) acknowledges support from the National Science Foundation grant 1912653. The sixth author (BSC) acknowledges financial support from the Brazilian National Council for Scientific and Technological Development (CNPq) in the form of a productivity grant (grant number 312951/2018-3).

We would like to thank Gerard Gorman at Imperial College London for his valuable feedback. We also acknowledge and appreciate the valuable feedback given by Wim Mulder, Amik St-Cyr, and Jorge Lopez from Royal Dutch Shell regarding full waveform inversion and finite elements.

We would also like to thank João Moreira for the generous comments and discussion.

This research has been supported by the Shell Brasil (grant no. ANP 20714-2), the National Science Foundation (grant no. 1912653), and the Conselho Nacional de Desenvolvimento Científico e Tecnológico (grant no. 312951/2018-3).

This paper was edited by James Kelly and reviewed by two anonymous referees.