This manuscript proposes an approach for geomodeling based on a deep neural network and a pseudo-labeling scheme to extract training data from borehole data. The method is tested on a shallow case study with a high density of boreholes, with a holdout validation and a cross-validation to assess model performance. The method is compared with three other approaches, although not quantitatively for one.
Overall the manuscript is much clearer than in the original submission, but understanding the training procedure and the validation remains difficult. As such, I have three main issues that need to be properly addressed before considering the manuscript for publication:
- Clarifying the training procedure, making sure that it can be reproduced from the manuscript alone.
- Describing the hyper-parameter tuning strategy used.
- Using a proper validation strategy, with a clearly defined test set and a cross-validation on all the wells from the training data to better estimate generalization error. This must include a proper comparison with the HRBF method, and tests on different settings that can be sub-sampled from the case study instead of introducing new case studies.
# Major comments:
Maybe I missed it but I couldn't find a response to my major comments from the first review. Some of them remain unanswered or only partially answered in the revisions, for instance:
- There is no mention of the exact architecture of the network, nor how it was developed and its impact on predictions. All that needs to be discussed in the paper.
- I am wondering whether the application is representative of real conditions. I am more used to deeper applications, where such high density of boreholes is impossible. This is essential since most implicit methods are developed for larger scale, deeper applications. If the study targets a specific setting (small-scale, shallow), then it needs to be more clearly stated. Similarly, the targeted application needs to be clarified (geotechnical applications?) and the goal of building a model as well. Why do you need to build a 3D model in the first place with so many data available? And do you really need such advanced method to build a 3D model? This is where the lack of quantitative validation is a big issue.
- There have been other studies on using deep learning for implicit modeling, e.g.:
https://doi.org/10.5194/gmd-2023-11
https://doi.org/10.5194/gmd-2022-290
How does this work compare to those? Both in terms of prediction errors and computational cost.
- Similarly, how does it compare with other modeling methods targeting quaternary deposits, e.g.: https://doi.org/10.3389/feart.2022.884075
- Overall there is a lack of discussion around the advantages and limitations of the proposed approach, especially compared with other approaches. There is a long list of papers mentioned in the introduction, but many are not really linked to this study, and most are not really discussed.
Although clarity has much improved compared to the previous version, the description of the training procedure still lacks key details (as such, I would be incapable of reproducing the approach based on the manuscript only). I'm still confused by the deterministic profiles of section 2.1 and by the prisms of section 2.3, because I can't figure out how they are actually used during training.
Section 2.3 mentions a two-step training procedure where the model is first trained on the true data, then on the true and pseudo data. I don't see that procedure represented in figure 3. I also don't understand how the pseudo data are exactly selected. It seems that predictions are made, and those with the highest accuracy are used as pseudo data in the second step. But how that accuracy is determined isn't clear.
How was hyper-parameter tuning performed? This is essential to get confidence in the results. Machine learning algorithms can be very sensitive to changes in hyper-parameter, so that using the default ones is not good enough.
Validation remains confusing at times, but it seems like it sill lacks robustness. Contrary to what is mentioned in the abstract, I don't see the HRBF in the results. It's unclear how the test set was determined, and why a proper cross-validation over all the wells except the test set wasn't done. In any case, cherry-picking wells for the cross-validation adds a considerable bias to the results.
A set of extra experiments with fewer data, for instance by randomly sampling 10, 50, and 100 wells, would give much more confidence that this use of predictions during training can actually generalize to other cases.
The lack of discussion of the advantages and limitations of the proposed approach is really a problem. I'm completely unconvinced that this approach would perform better on this case study than https://doi.org/10.5194/gmd-2022-290, https://doi.org/10.3389/feart.2022.884075, or any of the traditional implicit approaches, including the HRBF method. While you mention line 86 that "the issue of insufficient training data has yet to be adequately addressed", your case study is far from what I would consider insufficient in terms of data.
The introduction mentions tectonic structures, but how are those handled in this model? This could be a nice aspect to discuss for instance.
Another aspect interesting to discuss is whether similar approaches have been proposed by the deep learning community, and with which results (e.g., have later studies shown that it doesn't generalize so well?)
# Specific comments:
## Abstract
Line 17: What do you mean by "faces challenges related to uncertainty"? Implicit approaches are more automated than explicit ones, so it should be easier to manage uncertainty. However, they struggle to capture complex structures (e.g., Collon et al., 2016, 10.1190/INT-2015-0178.1).
Line 17: Kriging is one of the key approaches to implicit modeling, and kriging is the same as Gaussian processes in machine learning. So that opposition between explicit and implicit approaches on one side and machine learning approaches on the other is poorly substantiated. Machine learning is not a new tool for either explicit or implicit modeling. But using deep learning for explicit or implicit modeling is more recent, that would be true.
Line 18-19: It's not so much the use of machine learning (see my comment above, Gaussian processes work well in data-poor settings), but the use of implicit modeling (see Collon et al., 2016, 10.1190/INT-2015-0178.1, although their case study has considerably less data than this one, so I'm not sure this study can be called "data-poor").
Line 27: What does that mean "better [...] supports uncertainty analysis"?
Line 29: This is not a case study with sparse borehole data. Having a borehole every 23 m or so covering most of a 305 by 264 m domain is a high data density for a subsurface project.
## 1. Introduction
Line 37: It is a bit weird to end a list of common data with "other types of data". That list is limited in subsurface projects, so better be exhaustive. Maybe analog data from outcrops could be worth adding.
Line 54: What does "MLS" stand for?
Line 70: I don't understand the first point.
## 2. 3D Modelling Method Based on Deep Learning
Line 125: What if the TIN connect two intervals that shouldn't be connected? For instance two sandy intervals, but one is from a sedimentary channel, the other a crevasse splay? What would be the impact?
Figure 1: I'm still not sure I fully understand, so on the zoomed section, what are the points H1iPj mean? Are they just to illustrate the balance between different intervals? Or are they new data points created with the TIN? This remains quite confusing.
Line 170: It would be nice to have a figure describing the network's architecture (this could be done by updating figure 2).
Line 170: How can a user choose the right number of hidden neurons? How many layers are used? 4 like in figure 2? Why that choice? Is it robust or does it impact predictions significantly?
Line 195: Here a figure would help a lot.
Line 199: High accuracy based on what criterion?
Line 200: Is that done on the prisms mentioned before? Actually I'm still confused.
## 3. Experimental method and verification
Line 234: I'm still not sure what "unlabelled grids" mean. Are those related to the deterministic profiles?
Line 240: How was the test set determined?
Table 2: What are SAM and DL? And where is HRBF?
Figure 5 & 6: Is that only for SPDL? What about the other methods?
Line 258: Is that the cell size?
Line 266: How was the K-fold set up? By removing individual labels or entire wells?
Line 272: The procedure for the validation is not clear. You're only removing a few wells, so why not simply do a holdout validation (i.e., train only once to predict all the missing wells together directly)? And/or why not do a proper group K-fold cross-validation and predict all the wells?
Line 277: But in practice you won't be able to make such choices. This adds a considerable bias to your validation.
## 4. Discussion
Sections 4.1, 4.2, and 4.4: Those are not discussions, but new results that should be in section 3. The validation should be the same for all methods.
Line 311: I see no real reason for that. The goal of this paper is to predict geological formations, and the HRBF method should be compared on the same ground.
Line 329: A visual comparison is not robust enough for this. You need to add a quantified validation, so include the HRBF method in section 3.
Line 339: That's just untrue, erosive structures such as channels will lead to abrupt changes. You actually cannot say whether the HRBF method or SPDL perform better based on such weak comparison based on unsupported claims.
## 5. Conclusion
Line 411: That claim only holds is the both algorithms were properly tuned, which isn't mentioned in the paper.
Line 411: What about other indicators than accuracy? In imbalanced cases, accuracy is actually a poor indicator of performance, since a good accuracy can be achieved by predicting the major classes only.
Line 415: But other approaches to implicit modeling can capture uncertainty, how does you method compare to those? |
