Electrical resistivity tomography (ERT) is a broadly accepted geophysical method for subsurface investigations. Interpretation of field ERT data usually requires the application of computationally intensive forward modeling and inversion algorithms. For large-scale ERT data, the efficiency of these algorithms depends on the robustness, accuracy, and scalability on high-performance computing resources. In this regard, we present a robust and highly scalable implementation of forward modeling and inversion algorithms for ERT data. The implementation is publicly available and developed within the framework of PFLOTRAN, an open-source, state-of-the-art massively parallel subsurface flow and transport simulation code. The forward modeling is based on a finite-volume discretization of the governing differential equations, and the inversion uses a Gauss–Newton optimization scheme. To evaluate the accuracy of the forward modeling, two examples are first presented by considering layered (1D) and 3D earth conductivity models. The computed numerical results show good agreement with the analytical solutions for the layered earth model and results from a well-established code for the 3D model. Inversion of ERT data, simulated for a 3D model, is then performed to demonstrate the inversion capability by recovering the conductivity of the model. To demonstrate the parallel performance of PFLOTRAN's ERT process model and inversion capabilities, large-scale scalability tests are performed by using up to

Direct current electrical resistivity tomography (ERT) is one of the key geophysical methods for shallow-subsurface investigations, having applications in areas such as groundwater

ERT data simulation is performed by solving the electrostatic Poisson equation. Numerical methods are needed to solve the governing equation in arbitrary 3D conductivity structures. The most used numerical methods for 3D ERT modeling are the finite-difference (FD) method

The finite-volume (FV) method can also be used effectively to account for complex geometries

On the other hand, ERT data inversion is a nonlinear optimization problem to minimize a cost function that represents a measure of the difference between observed and simulated ERT data. For large-scale 3D ERT data, the inversion is often performed iteratively by linearizing the optimization problem at each iteration, calculating a model update, updating a conductivity model, and subsequently simulating ERT data for the updated model to examine the new cost function. The process continues until the cost function reduces to a predefined tolerance level. The model update is typically calculated using a gradient-based local optimization method, e.g., the steepest-descent, conjugate-gradient, or the Gauss–Newton method

For large-scale 3D ERT problems with tens of millions of degrees of freedom (DOFs), forward modeling and inversion are computationally demanding jobs and may need hours, if not days, of computing time if the underlying algorithms are not properly implemented to scale well on high-performance computing (HPC) resources or supercomputers. For such datasets, it is computationally infeasible to use desktop-based ERT data processing software without exploiting HPC resources. Recently, a few open-source 3D ERT modeling and inversion codes have been developed to handle moderately sized ERT problems composed of up to a few million DOFs, e.g., ResIPy

In this paper, we present an implementation of massively parallel 3D ERT modeling and inversion algorithms. The forward modeling is based on the FV method, and inversion employs the Gauss–Newton method.
The algorithms are implemented within PFLOTRAN (

The paper is structured as follows: we first outline a detailed background theory on the ERT forward modeling including the governing Poisson equation and FV method to solve it numerically. We then describe the ERT inversion by defining it as a nonlinear optimization problem and the Gauss–Newton method to find a minimizer of the problem along with details on computing the Jacobian matrix. Thereafter, we discuss the parallel implementations of the modeling and inversion algorithms. We subsequently benchmark numerical results computed using PFLOTRAN against analytical solutions for a layered (1D) earth model and numerical solutions for a 3D earth model. Next, a 3D inversion result is presented to illustrate the inversion capability. Scalability tests are then performed to demonstrate the parallel performance of the PFLOTRAN ERT process model before discussing the results and drawing our final concluding remarks.

Forward modeling is a way of simulating ERT data for any given 1D, 2D, or 3D electrical conductivity models of the earth. For 1D conductivity models, the electrical potentials generated by an induced point current source can be obtained by using analytical or semi-analytical methods

The ERT forward modeling is governed by the following electrostatic Poisson equation:

If side and bottom boundaries

To simulate

The FV method is implemented to compute

Two adjacent FV cells

Let us consider the governing Poisson equation (Eq.

By applying the Gauss divergence theorem in Eq. (

If the

Using a two-point flux approximation for each face term in Eq. (

The conductivity at the face,

The value of the flux at the face

Substituting Eqs. (

The assembly of the local coefficient matrix into a global system, upon carrying out the summation in Eq. (

ERT inversion is a procedure of estimating electrical conductivity of the earth from a set of observed ERT data and/or some imposed constraints. The ERT inverse problem is a nonlinear optimization problem. For large-scale 3D ERT data, the inversion is often performed iteratively by linearizing the problem at each iteration. This section details our approach of performing inversion of ERT data.

We pose the inverse problem as a nonlinear minimization problem of finding a conductivity model

A least-squares method is used to minimize

To find a minimizer

The matrix

The solution of Eq. (

In our implementation, the Gauss–Newton iteration usually starts with an average apparent conductivity model as an initial model and continues until the data misfit drops below a predefined tolerance level (defined by a chi-squared data misfit

Calculating

An adjoint state method is used to effectively compute the Jacobian matrix. Let

Wenner electrode configuration: four electrodes

Using the expression derived in Appendix

where

Assuming

Equation (

The parallel implementation strategy for the ERT forward modeling and inversion capabilities follows the existing PFLOTRAN’s strategy for flow and transport process models. Forward ERT modeling is implemented by extending PFLOTRAN's hierarchy of Fortran classes and data structures to create a new ERT process model that constructs and solves the linear systems of equations described in Sect.

To execute simulations in parallel, domain decomposition is employed to distribute the discrete computational domain across (computer) processes using a division along the three principal axes for structured grids (calculated by PETSc or explicitly defined by the user), or in the case of unstructured grids, the ParMETIS parallel graph partitioner

Efficient partitioning and the overlapping of communication with computation (e.g., calculating floating point operations while data are being transferred between processes) helps improve parallel performance. The latter helps mask communication costs. The goal is to maximize the time spent in computation and minimize interprocess communication and enforce

PFLOTRAN's parallel file I/O is through HDF5 (Hierarchical Data Format version 5), a platform-independent binary file format that leverages MPI-IO operations to read/write a single file using multiple (potentially thousands of) processes

To examine the accuracy of the PFLOTRAN ERT process model, simulation results are compared to solutions obtained using well-established analytical and numerical methods. Two cases are considered for the comparison: three-layer earth and 3-D earth resistivity models. All benchmarking simulations were performed on the Deception supercomputer housed at the Pacific Northwest National Laboratory. It is composed of

Let us consider the layered earth or 1D model shown in Fig.

Vertical cross section of a three-layer earth resistivity model used for benchmarking PFLOTRAN results against the analytical solutions.

Apparent resistivity

To simulate the VES data, the model is discretized using a mix of uniform and non-uniform cell sizes in the

The simulated VES results are compared to analytical solutions obtained from SimPEG

In the previous example, PFLOTRAN accuracy for simulating ERT responses was validated for a simplistic 1D earth model. In the real world, however, the earth models are 3-D with varying degrees of heterogeneity. Therefore, it is important to validate the accuracy of numerical results computed using PFLOTRAN against results computed using well-established methods for 3D models.

Three-dimensional earth resistivity model composed of two anomalous blocks embedded in a homogeneous background. ERT profiling data are simulated along a survey profile at

We consider the 3D model shown in Fig.

Apparent resistivity

The main computational domain is discretized with a uniform grid spacing of

Figure

To examine the efficiency of PFLOTRAN to invert ERT data, we consider a model modified from Fig.

The simulated ERT data were then inverted using PFLOTRAN starting with a model having a half-space conductivity of

Inversion result of the simulated ERT data for a model modified from Fig.

A measure of data misfit as the inversion progresses:

Histogram of data-fit error, which is defined as ratio of the residual resistance (difference in the observed and predicted resistances) and observed resistance expressed as a percentage.

Figure

The initial chi-squared data misfit

Scalability tests demonstrate parallel performance or how efficiently a simulator utilizes additional processes to either speed up a simulation of fixed problem size (strong scaling) or maintain speed as the problem size and number of processes increase proportionally (weak scaling). For ideal strong scaling, the speedup is proportional to the increase in the number of processes (e.g., doubling the number of processes cuts the run time in half), while ideal weak scaling results in an unaltered or constant run time. In either (ideal) case, the additional computing capacity is efficiently utilized. Ideal strong scaling is often termed linear speedup, whereas superlinear speedup is better than ideal (e.g., doubling the number of processes reduces the run time to less than half). Superlinear performance is not uncommon, as access to an increased cache (efficient, high-performance memory on the processor core) can generate superlinear speedup.

As discussed earlier, there are two computationally intensive steps to invert ERT data: forward modeling and Jacobian computation. To demonstrate the parallel performance of PFLOTRAN for these two steps, large-scale strong scaling tests are reported in this section by using up to

The three-layer earth model in Fig.

Test matrices obtained by rediscretizing the three-layer earth model in Fig.

PFLOTRAN scalability for simulating ERT data for the model in Fig.

Considering the six test matrices above, we performed the forward modeling scalability test on Theta for simulating ERT data for

For all problem sizes, the wall-clock times flatten at higher numbers of processes. This phenomenon is expected as the amount of work per process (DOFs per process) decreases at higher process counts and the cost of communication between processes begins to dominate.
The M19 performance begins to deviate from ideal at

Another factor limiting parallel performance is the limited scalability of linear Krylov solvers at high process counts.

To test the scalability of PFLOTRAN for the Jacobian computation, we consider a Jacobian matrix computed for inverting the ERT data in the “Inversion result” section (Sect.

PFLOTRAN strong scaling for computing the Jacobian for the ERT inversion example in Fig.

The Jacobian scalability test is performed on the Deception supercomputer. Figure

For large-scale ERT inversion problems, the linear scaling for computing

In the past 20 years, a few open-source 3D ERT modeling and inversion codes were developed to handle moderate-scale ERT problems (e.g., ResIPy, SimPEG, pyGIMLI, and E4D). Although some of these codes can practically tackle most present ERT modeling and inversion problems having less than

Due to the efficient parallel performance of PFLOTRAN observed above, a large-scale ERT simulation problem with

Finally, there were two key motivations for implementing ERT capabilities in PFLOTRAN. The first motivation was to utilize the massive parallel infrastructure implemented within PFLOTRAN for the flow and transport simulations as illustrated by

The second motivation was to have a native implementation of ERT modeling and inversion capabilities along with flow and transport simulations. The new capabilities will allow us to perform fully coupled hydro-geophysical simulations and inversion. The idea behind the joint hydro-geophysical simulation is to use known flow and transport parameters, such as porosity and intrinsic permeability, and simulate for hydrological variables, e.g., liquid saturation and solute concentration. Using petrophysical relationships (e.g., Archie's law or its modifications;

Furthermore, there are growing interests in joint hydro-geophysical inversion, i.e., estimating the distribution of subsurface flow and transport parameters, from time-lapse ERT data; see, e.g.,

Two new capabilities – forward modeling and inversion of geophysical ERT data – are developed within PFLOTRAN, a massively parallel open-source, state-of-the-art, multi-phase, multi-component subsurface flow and transport simulation code. The capabilities are accurate, robust, and highly scalable on HPC platforms. The accuracy of the forward modeling capability was demonstrated for layered earth and 3D earth conductivity models by comparing our numerical results against well-established analytical and numerical results. The average relative differences stayed within 1 %, demonstrating the high degree of accuracy of the PFLOTRAN ERT process model. The inversion capability was demonstrated for a 3D synthetic case by recovering the conductivity of the model. Large-scale scalability tests illustrated that the forward modeling for models with 100

Let us assume that

Therefore, the

Using Eq. (

Taking the derivative of Eq. (

Furthermore, taking the derivative of Eq. (

If

Using Eq. (

As

If the same interpolation scheme is used for placing the current electrodes within the discretized cells and recording the potential at electrodes from the cell potentials, we will have

The source code, input file, and results presented in this paper are based on PFLOTRAN version v4.0. It can be downloaded from its Git repository at

PFLOTRAN is available under the GNU Lesser General Public License (LGPL) version 3.0. The dependencies listed here are available under compatible open-source licenses except for ParMetis, which is free for research and education purposes by non-profit institutions and US government agencies.

PJ implemented the ERT modeling and inversion capabilities with contributions from GEH. PJ performed the tests, analyzed the results, and prepared the figures. TCJ supervised the study. PJ wrote the paper with contributions from GEH, and all co-authors read and helped with editing.

The contact author has declared that none of the authors has any competing interests.

Publisher’s note: Copernicus Publications remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

We gratefully acknowledge computing resources available through Research Computing at PNNL and Argonne Leadership Computing Facility (ALCF). ALCF is a DOE Office of Science User Facility supported under contract DE-AC02-06CH11357. Finally, we would like express our gratitude to chief executive editor David Ham, topical editor Adrian Sandu, and two reviewers Mark Everett and Michael Tso, for their careful reading, suggestions, and constructive comments that significantly helped to improve the paper.

This study is a part of the INSITE (INduced Spectral Imaging for The Environment) Agile investment at Pacific Northwest National Laboratory (PNNL). It was conducted under the Laboratory Directed Research and Development Program at PNNL, a multi-program national laboratory operated by Battelle for the U.S. Department of Energy (DOE) under contract DE-AC05-76RL01830.

This paper was edited by David Ham and reviewed by Mark Everett and Michael Tso.