Dear Referees and Editor,

We thank you very much for your time reviewing the manuscript and providing valuable feedback to improve it. The manuscript will be carefully revised following your comments/suggestions. Please see the following for our responses to your comments.

Reviewer 1:

Dear Dr. Mark Everett,

We appreciate your valuable remarks on the manuscript. We will carefully revise the manuscript and provide a detailed response during the submission of our revision. Here are our responses to your comments:

1. main claim warranting publication is scalability

There are two key aspects of the manuscript: the development of an open-source massively parallel electrical resistivity tomography (ERT) modeling and inversion capability within PFLOTRAN and demonstrating its scalability on supercomputers. The first aspect is important because, to our knowledge, no open-source massively parallel code exists that can solve large-scale (with up to a hundred million degrees of freedom or DOFs) ERT problems. The second part is indeed illustrating its strong scaling on supercomputers (linear speedup when the number of processes employed is increased on a fixed problem size). This allows us to solve large-scale ERT simulation problems in less time.

2. sections 2-3, 5-6 are not new and should be placed in appendices

Section 2 provides brief info on the forward modeling along with a detailed description of the finite-volume (FV) method. To the best of our knowledge, the FV method, in the context of ERT modeling, has not been discussed before, and this section will be beneficial for the journal’s readers. Section 3 provides details on the ERT inversion scheme, and we agree that this section may not be providing significantly new information. However, the section details our approach to inversion, which is essential for understanding the code and its functionality. In the end, we will move some of the details from this section to an appendix without which the section can still be explained.   

Sections 5-6 provide benchmarking results for the modeling and inversion. Since the ERT modeling and inversion capabilities are newly developed from scratch while exploiting the infrastructure of PFLOTRAN (e.g., meshing, partitioning, and linear system solvers), it’s crucial to benchmark the numerical results from the developed capabilities to demonstrate the accuracy of the results (within a tolerance level) before presenting the scalability tests. These results are new and have never been presented/published before. Therefore, we will keep these sections in the main text.    

3. sections 4 and 7 need to be greatly expanded, and a discussion section added

We thank you for the comment. We will provide more details in Section 4 (Parallel Implementation) as appropriate since most of our parallel implementation strategy for ERT is based on the general parallel strategy of PFLOTRAN code. Relevant papers on the general PFLOTRAN parallel implementation strategy will also be cited along with highlights of any differences for the ERT parallel implementation. Section 7 will be expanded, and a discussion section will also be added to the revised copy.  

4. too much undefined jargon!

There might be some undefined terms, specifically for parallel implementation. Though most of these words are well-known terms within the parallel computing community, we do recognize that these words may not be commonly used within the general geoscience community. So, we will simplify the explanation and provide as much details as possible wherever necessary.  

5. for details see comments on pdf

Thank you. We will provide responses to these comments during the revision of the manuscript.

Reviewer 2:

This article summarizes the extension of PFLOTRAN, a popular massively parallel code for subsurface flow and transport simulations for electrical resistivity tomography (ERT) modeling. ERT is one of the most popular geophysical methods.

The article is pitched to focus on the parallel capabilities and good scaling for forward modeling and Jacobian computation.

Dear Dr. Michael Tso,

We greatly appreciate your constructive feedback on the manuscript. Point-by-point responses to your comments are the following. The revised copy of the manuscript will include your suggested changes.  

1. The authors have mentioned another well-established massively parallel code, E4D (L65). What is the motivation of including ERT capabilities in PFLOTRAN (given users won’t typically expect geophysics capabilities in a flow and transport code). How different is the current ERT implementation different from E4D’s? Was the goal to reproduce them as close as possible? Are there any plans to retire E4D?

Thanks for this question. One of the main motivations for implementing the ERT module in PFLOTRAN is to have a native implementation of ERT modeling and inversion capability so that coupled flow, transport, and ERT simulations (and inversion) can be performed. A demonstration of such capability will be presented in our forthcoming paper(s). As explained in the manuscript, ERT implementation in PFLOTRAN is based on a structured-mesh finite-volume discretization, whereas in E4D (Johnson et al. 2010) it is based on an unstructured mesh finite-element discretization of the governing differential equation. In addition, the finite element implementation in E4D solves for degrees of freedom at the nodes (cell vertices) whereas the finite volume implementation for ERT in PFLOTRAN solves for unknowns at the cell center, which is aligned with the flow and transport degrees of freedom, which are at cell centers. Moreover, E4D has additional capabilities, e.g., real-time and time-lapse (4D) inversion and metallic infrastructure modeling, which are not implemented in PFLOTRAN. We do not have any plans to retire E4D software which will continue to be used for standalone ERT modeling and inversion problems. The inclusion of ERT simulation in PFLOTRAN completes the capabilities necessary for native hydro-geophysical modeling and inversion within PFLOTRAN, which will be described in forthcoming papers. For example, this development is a necessary step toward using time-lapse ERT monitoring data to estimate hydrogeologic properties through joint inversion within PFLOTRAN. These details will be added to a new Discussion section. 

2. I understand the authors focus this work on ERT modeling only but it is still relevant to mention PFLOTORAN-E4D (Johnson et al. 2017), a now discontinued PFLOTRAN feature (I.e. the HYDROGEOPHYSICS mode) to call E4D for coupled hydrogeophysical simulations. I think it will help explain the motivation behind this work.

We appreciate the comment. PFLOTRAN-E4D joint implementation was based on getting flow and transport parameters from PFLOTRAN using a structured mesh then these parameters were needed to be interpolated on an unstructured tetrahedral mesh for E4D simulations. This process was cumbersome and has the potential for errors. On the other hand, the new implementation of ERT modeling and inversion natively in PPFLOTRAN avoids using any kind of interpolation because both flow & transport and ERT simulations use the same simulation grid. We will add a few sentences in the revised manuscript for more clarification.   

3. Are all the inversion and output options from E4D available in this new implementation?

As we wrote in response to your comment #1, E4D has more advanced ERT modeling and inversion capabilities than PFLOTRAN. For example, E4D can be used for real-time and time-lapse ERT inversion as well as metallic infrastructures can be modeled, however, PFLOTRAN lacks these advanced capabilities. Nevertheless, PFLOTRAN can perform coupled flow, transport, and ERT simulations natively and solve ERT problems with hundreds of million unknowns (DOFs) in the linear system.    

4. RC1 suggests more details on parallel implementations. I think if it is not too different from PFLOTRAN in general, then a simple statement to state this fact will be sufficient. Any differences in parallel implementations should be highlighted.

Indeed, our parallel implementation of ERT capabilities is not too different from the existing PFLOTRAN’s parallel implementation strategy for the flow and transport simulations. We will clarify this in the revised manuscript and highlight any differences in the ERT parallel implementation along with appropriate citations of published papers on the general parallel implementation strategy used in PFLOTRAN.

5. This work boasts the capability to model problems with very large DOFs. It will be helpful to discuss what practical problems may benefit from such capabilities. A modelling example with field data will be helpful. A comment on usability will be helpful too. For example, it essentially reads a PFLOTRAN input deck cards, so users familiar with PFLOTRAN can immediately take advantage of this new feature.

The capability to solve ERT simulation problems with hundreds of million DOFs is mainly motivated by the fact that significant efforts have been made recently in building massive supercomputers with hundreds of thousands or some even millions of computing processors/cores. But most of the existing ERT modeling and inversion codes are still restricted by using only a few hundred or thousand processors, and as a result, are restricted to solving problems with only up to about ten million DOFs. To avoid such limitations, we needed to implement massive parallelization of ERT capabilities, and PFLOTRAN, already having such parallelization infrastructure for the flow and transport simulations, was a natural fit to achieve this goal. Thanks for your comment on the usability, we will add a few sentences on the same following your comment.  

6. The example model runs and scalability tests are sufficient to illustrate the points made in the paper.

We thank you for this comment.

7. I understand the public version/main branch of PFLOTRAN v4.0 does not include ERT capabilities. Make a note on which branch is copied to the repo.

Sorry for any confusion! The public version of PFOTRAN v4.0 does include the newly implemented ERT capabilities. Please read the Code Availability section for further details on how this version can be checked out and compiled from its public Git repository. Moreover, a snapshot of PFLOTRAN v4.0 is also uploaded to Zenodo along with all the examples presented in this paper. The latest master/main branch also includes all ERT capabilities presented in the manuscript, but it is likely that examples uploaded at Zenodo may not properly run using the latest master branch due to changes made to the input format as a result of the ongoing development of the coupled flow, transport, and ERT capabilities. So, we would recommend using PFLOTRAN v4.0 for reproducing the presented results and using its functionalities for similar applications.  

Finally, we would like to thank Dr. Mark Everett and Dr. Machael Tso once again for their constructive feedback and comments.

Best regards,

Piyoosh Jaysaval, on behalf of all co-authors

Johnson, T. C., Versteeg, R. J., Ward, A., Day-Lewis, F. D., and Revil, A.: Improved hydrogeophysical characterization and monitoring through parallel modeling and inversion of time-domain resistivity and induced-polarization data, GEOPHYSICS, 75, WA27–WA41, https://doi.org/10.1190/1.3475513, 2010.