Dear Dr Gelbrecht,

Thank you for your revised submission, please accept my apologies for the time it has taken me to come back to this. I’m afraid that at this stage this manuscript is a fair way from publishable.

The first key issue, which was somewhat identified by reviewer 2 and not really addressed in the revised manuscript, is that it’s not at all clear what it is you mean by differentiable programming, and hence what the change to ESM development that you are proposing actually constitutes. This is despite the presence of section 3, which is titled “differentiable programming”. It seems from that section that by “differentiable programming”, you mean some programming paradigm in which AD works well, and you cite machine learning frameworks as examples of this. So far so good, but we don’t get to learn what it is about a programming model that makes AD work well and what the difference is between that and the programming models which have hitherto generally been used for ESMs. Indeed, in section 4 you include the example of TAF which has been used, albeit with massive manual intervention primarily by Patrick Heimbach, to adjoin the MITGCM among others. You claim that there is something about modern AD systems which makes this manual intervention unnecessary, but we never get to hear what it is.

There are two issues with this failure to define what the programming paradigm you are arguing for actually is. Your audience in GMD are geoscientific model developers who on the whole are familiar with how ESMs are written, may have some idea about how AD works, but who aren’t on the whole ML people. I don’t currently think that an ESM developer reading this manuscript would understand what you are advocating that they do. The second, and more fundamental, issue is that I don’t think you’ve demonstrated that you know what the change required is either, and you certainly haven’t articulated it.

I think that your basic premise has merit: that a change in programming paradigm would make ESMs and ESM components more amenable to AD, and that this would be a good thing for the reasons that you give. I should declare an interest at this point: as I am sure you are aware, I am one of the authors of Farrell et al (2013) and am the leader of the Firedrake project. I therefore do have an opinion on exactly what the defining characteristics of what might be called differentiable programming are, which I will give here. In order to turn this manuscript into something publishable, you don’t have to agree with me, but you do need to articulate your argument about what differentiable programming is and to what extent it is applicable to ESMs.

There have always been two essential forms of AD: source-to-source translation, and operator overloading and taping (this is discussed in depth in Naumann, 2011). In essence the distinction is one of static analysis versus dynamic recording. To grossly over-simplify, the problem with the former is that it lacks robustness for more than rather simple operations, while the latter incurs the time and memory overhead of recording and running the tape. In fact, all real systems are some sort of hybrid, because the tape is still made up of base operations and the forward or reverse mode derivative of each base operation is a static analysis task. If a taping approach is applied at the level of individual floating point operations, for example by using a dual numbers approach in C++, the overhead is likely to swamp the original computation cost. The key insight in Farrell et al. (2013) and also in ML frameworks such as Pytorch and TensorFlow (interestingly, earlier versions of TensorFlow used fully static analysis but they gave up in favour of taping) is that taping is a much more performant approach if your base operators are not individual floating point operations, but rather much larger computational units. In ML frameworks these are operations over large tensors. In Farrell et al. (2013), the high level abstraction provided by the finite element method and embodied in FEniCS (and later Firedrake) meant that the taped items are whole PDE solves.

I think that this means that “differentiable programming” refers to programming models in which the base operations are simple enough to be differentiable using static analysis, but comprise enough floating point operations that the overhead in the tape operations is at least tolerable (it may even be negligible).

If this is the general characteristic of differentiable programming (once again, feel free to argue differently) then we can come to consider what a differentiable programming model suitable for ECMs might look like. Reviewer 2 gave you some examples of existing geoscientific models which are written in differentiable programming models. You argued that considering them was out of scope because they were all finite element models. In fact Devito is a finite difference framework and Reviewer 2 could have mentioned Dedalus (https://dedalus-project.org) which is a spectral framework with a lot of similarities to the other models there. The key point is that Firedrake and FEniCS, Devito, and Dedalus all provide programming models in which high level abstractions are provided for the PDEs being solved. Where automatic adjoints for these tools have been created, they tape at the level of the abstraction, not the primitive operations that implement the abstraction. It’s not a coincidence that these are predominantly finite element and spectral models, because these are discretisations that for which more complete high-level abstractions exist and hence for which a differentiable programming approach is readily applicable.

Finite difference and finite volume methods, on the other hand, lack a general abstraction, especially for advection operators. Indeed, Devito is specialised for tomography applications in which only wave equations are solved: it doesn’t have a solution to this problem. For example Clawpack, which in some senses is a high-level abstraction for finite volume, still requires Riemann solvers to be provided in low-level code (in that case in Fortran, but the particular language doesn’t matter). If finite difference and finite volume models require bespoke low-level code for core parts of the model, then only the classical AD approaches (e.g. TAF) will be available for them. The one example you cite, Souhar et al (2007) uses Tapanade, which is a similar tool to TAF. As you correctly point out in section 4, those systems have limitations that have prevented their widespread adoption. However, the final statement in section 4 “modern, more capable AD systems in combination with machinery originally developed for ML tasks promise substantially greater benefits” is currently unjustified and I am not actually convinced that it’s true to the extent that it would enable the efficient adjoining of finite difference and finite volume ESM components. You certainly haven’t explained what it is that has changed that makes more modern AD systems more capable.

This brings us to your dismissal of reviewer 2’s suggestion that you need to discuss existing differentiable geoscientific models. You claimed that it wouldn’t be in scope to discuss them because ESMs use a much wider range of discretisations. My response is that this misses the point: the existing differentiable programming tools for geoscience are predominantly finite element because that (potentially alongside spectral, which has very similar characteristics) is the discretisation to which the differentiable programming paradigm is applicable. I think there are two ways you could remedy the manuscript in this regard. You could discuss the limitations of differentiable programming as currently understood, in which context the geoscientific models for which it does work are highly relevant. Alternatively, you could provide a more detailed argument about a different conception of what differentiable programming is, and provide an explanation of what has changed such that the previous limitations have been overcome. In the latter case I would expect you to be able to point to a case where whatever your definition of differentiable programming is has been successfully applied to a finite difference or finite volume flow model (preferably an ESM component, but that’s not really necessary in order to prove your point).

Finally, I think you have been too blasé about the challenges imposed by flow control. You have stated that AD systems can differentiate through flow control. This is true in the sense that the tools produce a result. However, that result is only a locally consistent gradient: if the controls are changed such that the forward model would take a different branch, it’s far from automatic that the AD system’s gradient (adjoint or TLM) will transition to the derivative of the other branch. Indeed, with a taping approach it will most certainly not. This also leaves aside the question of what the right gradient is in the vicinity of the branch. This is neither an easy nor a trivial point, a fact which has been recognised for decades (see, for example, Bao and Kuo, 1995). One can work around it in a manual way by defining a function which is some approximation to the gradient, or by changing to parametrisations that are at least weakly differentiable. However, those approaches undermine your argument that the process needs to be automated. This is a limitation which needs to be mentioned.

Concretely, based on the above, what I think you need to do is the following:

1. Rewrite (and I mean comprehensively, not just add a couple of sentences) section 3 so that it has a clear definition of what you are arguing differentiable programming is. This must be written in such a way that an ESM developer will understand the concrete difference between what you are claiming as differentiable programming and the conventional AD approaches (e.g Tapanade and TAF) which have been known in the field for decades. The rest of the manuscript should be adapted to be consistent with what you write here.

2. Actually make the case for what aspects of ESMs can and can’t be accommodated by your definition of differentiable programming. This needs to be much more concrete than the current assertions, which come pretty close to just claiming that everything works but do not provide any explanation as to why things that have long been known to be problematic are apparently no longer so. This particularly applies to parametrisations containing switches, and advection schemes in finite difference and finite volume models.

3. You do need to at some level mention the state of the art in differentiable ESM components. If you agree with my analysis above then this could consist of discussing the packages that reviewer 2 raised. If you disagree with my analysis then I’d expect you to be able to point to some finite difference and/or finite volume geoscience models written in differentiable programming paradigms, or possibly to be able to point to a reason other than my reasoning as to why finite element seems to dominate this field.

Yours,

David

Bao, J.W. and Kuo, Y.H., 1995. On–off switches in the adjoint method: Step functions. Monthly weather review, 123(5), pp.1589-1594.

Naumann, U., 2011. The art of differentiating computer programs: an introduction to algorithmic differentiation. Society for Industrial and Applied Mathematics.

Dear Dr Gelbrecht,

Thank you for your revised submission. I agree that this is substantially addresses my concerns and is now ready for publication.

I have just a couple of technical corrections:

1. Please correct the spelling of dolfin-adjoint throughout. The original dolfin system, a core component of FEniCS, was written by Anders Logg who is Swedish. Dolfin therefore is therefore spelt in the Swedish rather than the English way.

2. The statement at line 347 "However, it has only limited interfacing with ML workflows and libraries." is not really true. Firedrake supports the embedding of ML models, including differentiation through dolfin-adjoint (https://doi.org/10.48550/arXiv.2111.00945). The converse embedding of Firedrake in PyTorch has also been done but the accepted conference paper is yet to appear, so there's no way you could have known that.

Again I should declare my interest since I am an author of those papers.

Yours,

David

Please consider the technical corrections as suggested by the topic editor.