the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Metrics for assessing Linear Inverse Problems: a case study of a Trace Gas Inversion
Vineet Yadav
Charles E. Miller
Abstract. Multiple metrics have been proposed and utilized to assess the performance of linear Bayesian and geostatistical inverse problems. These metrics are mostly related to assessing reduction in prior uncertainties, comparing modeled observations to true observations, and checking distributional assumptions. These metrics though important should be augmented with sensitivity analysis to obtain a comprehensive understanding of the performance of inversions and critically improve confidence in the estimated fluxes. With this motivation, we derive analytical forms of the local sensitivities with respect to the number of inputs such as measurements, covariance parameters, covariates, and forward operator or jacobian. In addition to local sensitivity, we develop a framework for global sensitivity analysis that shows the apportionment of the uncertainty of different inputs to an inverse problem. The proposed framework is applicable to any other domain that employs linear Bayesian and geostatistical inverse methods. We show the application of our methodology in the context of an atmospheric inverse problem for estimating urban GHG emissions in Los Angeles. Within its context, we also propose a mathematical framework to construct correlation functions and components of uncertainty matrices from a precomputed jacobian that encompasses nonstationary structures.
 Preprint
(8350 KB) 
Supplement
(6269 KB)  BibTeX
 EndNote
Vineet Yadav et al.
Status: closed

RC1: 'Comment on gmd202289', Peter Rayner, 03 Jul 2022
This paper presents a series of quantities that can be derived from linear inverse theory. Put roughly they are the similarity of footprints (nearly the independence of rows of the Jacobian), the local sensitivity of the result to various inputs and finally a global sensitivity using a firstorder Taylor expansion with respect to all inputs. The metrics are potentially useful and some are, to my knowledge, novel. The paper is potentially in scope for AMT though I think it needs more work to make it more relevant to likely readers.
My first problem with the paper is its title. the word "assessment" suggests some comment on the quality or robustness of an inversion. The authors don't do that and it's not clear from the paper that the generated metrics can do it. I'm unclear, for example, what new information is provided by the overlap of footprints. It might well mean that parts of the control space are undersampled by the observations but the posterior uncertainty already tells us this. For the linear Gaussian case the posterior uncertainty can be calculated without measurements.
Likewise the sensitivity of the posterior estimate to the value of a given measurement is potentially useful as a warning flag for measurements that might have undue control on the outcome but it's not really developed. the global sensitivity analysis, which allows consideration of all inputs to the linear inverse problem, is potentially more interesting but again is not developed beyond generation of the firstorder expansion. The example presents a good opportunity to demonstrate application of these methods but this is not taken far beyond calculation of the diagnostics.
I see two possibilities for the paper:
1) Repackage it as a technical note focusing on the calculation of the diagnostics
2) Extend the work to generate diagnostics of overall inversion performance, probably focusing on robustness.Specific Comments:
L130: What is a footprintinduced probability distribution?
L190: The definition of the averaging kernel is true but this is an odd motivation for it, much better below when contemplating sensitivity of result to prior
Eq. 19: it's worth noting that this sensitivity is very close to the proportional uncertainty reduction A P^1 and hence the averaging kernel. By the way I thank the authors for making me think hard enough about the relationship between AK, DOFS and uncertainty reduction to finally get an intuitive sense of it
L275: When commenting on covariances between H, Q etc we should also note that constraints like conservation of mass introduce strong covariances within the parameters of H. Covariances can only occur on physically plausible manifolds. This is a profoundly understudied problem in transport modelling and there is probably great insight to be borrowed from Numerical Weather Prediction.
L431: not sure what the authors mean by aggregation error here. If they're truly commenting on temporal aggregation error they should cite DOI:10.5194/acp1134432011.Citation: https://doi.org/10.5194/gmd202289RC1 
AC1: 'Reply on RC1', Vineet Yadav, 07 Sep 2022
The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd202289/gmd202289AC1supplement.pdf

AC1: 'Reply on RC1', Vineet Yadav, 07 Sep 2022

RC2: 'Comment on gmd202289', Anonymous Referee #2, 15 Jul 2022
This paper presents methods for sensitivity analysis of the solution of the inverse problem to various input parameters, keeping in mind the cost associated with the sensitivity computations. This is important because it provides the user with some understanding of the sensitivities and may lead to better prior models or measurement strategies. The methodology is illustrated in the case of an atmospheric inverse problem. The computational costs are significant for such problems, even for computing the MAP estimate, so it is important to derive efficient methods for sensitivity analysis.
Major concerns:
 Contributions: I couldn’t fully understand the contributions of this paper. Is it claiming to be developing new methods for inverse problems, or applying existing techniques to the case study? The methods for local sensitivity analysis involving derivatives appear to be a special case of the Hyperdifferential sensitivity analysis for linear inverse problems and the GSA method is essentially DGSM (see end of review).
 Mathematical exposition: I understand that this is not an applied math journal, but the standard for exposition was well below this journal. The notation is not setup properly and inconsistently used. There were a lot of vague statements (I did not fully tabulate this list).
Specific comments:
 Title: I think working in the word sensitivity is better here than “assessing”.
 The abstract does not clarify, of what quantity (i.e. MAP estimate) the sensitivity is being computed. GHG is not expanded and Jacobian should be capitalized throughout. The word Jacobian (to me) is misleading because this is a linear problem – there are other Jacobians used in the paper.
 The introduction does not clearly list the contributions (as mentioned earlier) and is missing some references.
 Section 3:
 The inversion is not setup properly before 3.1.13.1.3 are explained. I essentially did not understand anything in these subsections. I am not sure what sets are being considered, how this relates to the inversion, etc. There is a small discussion at the end of 3.1.3 but it is referring to things that haven’t yet been defined.
 Section 3.2: Someone not familiar with this material will struggle since it has not been discussed properly. I suggest reorganizing this section bringing some of this material earlier.
 Notation/Writing: This is not consistent throughout the paper. Sometimes subscripts refer to sizes, other times they mean elements. Sometimes boldfaced, sometimes not. Sometimes lower/upper case. Sentences should not start with variables. When units are being described, the entries of vectors have units, not the vectors themselves.
 Line 157: it’s > its. This occurred in other places also.
 Line 169: it can be simplified using the notation below in 172.
 Line 173: One differentiates a function rather than an equation.
 When referring to equations, one typically puts a parenthesis, e.g. (10).
 Line 178: The inputs to the inverse problems are data and hyperparameters. The transport model is typically fixed, as is the drift matrix X. What does it mean to differentiate wrt them? Why is that useful in applications? Are these matrices functions of some hyperparameters?
 Line 185: The equation is not referred to as Lambda but one of the terms there is Lambda.
 Line 203: When differentiating a vector wrt a matrix, one should get a tensor. I think what is happening is that you are vectorizing X and then differentiating. But the notation is not clear. Same with line 227.
 Section 3.3: I agree that full GSA is very complicated. It’s not clear how this is GSA. Is this essentially derivative based global sensitivity? Once again I did not really understand what was being discussed.
 Section 3.4: The authors raise a good point here cautioning against sensitivity metrics since they are of different units but little has been done to address them. There are techniques in sensitivity analysis that the authors should consult (DGSMs and activity scores are some techniques, see Constantine and Diaz)
 Section 4: I didn’t understand the figures since sometimes the sizes of the quantities are not clear. Are entrywise sensitivities being plotted?
 Appendix: The notation was especially problematic here.
Returning to the major concerns: I don’t think the paper is novel in the methodology, but as an application to atmospheric inverse problems I think it has potential. In its current version, I don’t think it is worthy of publishing. In addition to the comments I raised, I would encourage the authors to think of the following questions to improve the utility. What is the computational cost of these approaches? Can they be computed when the forward model is not available entrywise? What are the challenges involved and how can they be efficiently implemented? What are the strengths and weaknesses of each approach? How does one compare sensitivities of different quantities (with different units) and rank the sources of sensitivity?
References:
 Sunseri, Isaac, et al. "Hyperdifferential sensitivity analysis for inverse problems constrained by partial differential equations." Inverse Problems 36.12 (2020): 125001.
 Sunseri, Isaac, et al. "Hyperdifferential sensitivity analysis for nonlinear Bayesian inverse problems." arXiv preprint arXiv:2202.02219 (2022).
 Sobol, I. M., and S. Kucherenko. "Derivative based global sensitivity measures." ProcediaSocial and Behavioral Sciences 2.6 (2010): 77457746.
 Constantine, Paul G., and Paul Diaz. "Global sensitivity metrics from active subspaces." Reliability Engineering & System Safety 162 (2017): 113.
Citation: https://doi.org/10.5194/gmd202289RC2 
AC2: 'Reply on RC2', Vineet Yadav, 07 Sep 2022
The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd202289/gmd202289AC2supplement.pdf
Status: closed

RC1: 'Comment on gmd202289', Peter Rayner, 03 Jul 2022
This paper presents a series of quantities that can be derived from linear inverse theory. Put roughly they are the similarity of footprints (nearly the independence of rows of the Jacobian), the local sensitivity of the result to various inputs and finally a global sensitivity using a firstorder Taylor expansion with respect to all inputs. The metrics are potentially useful and some are, to my knowledge, novel. The paper is potentially in scope for AMT though I think it needs more work to make it more relevant to likely readers.
My first problem with the paper is its title. the word "assessment" suggests some comment on the quality or robustness of an inversion. The authors don't do that and it's not clear from the paper that the generated metrics can do it. I'm unclear, for example, what new information is provided by the overlap of footprints. It might well mean that parts of the control space are undersampled by the observations but the posterior uncertainty already tells us this. For the linear Gaussian case the posterior uncertainty can be calculated without measurements.
Likewise the sensitivity of the posterior estimate to the value of a given measurement is potentially useful as a warning flag for measurements that might have undue control on the outcome but it's not really developed. the global sensitivity analysis, which allows consideration of all inputs to the linear inverse problem, is potentially more interesting but again is not developed beyond generation of the firstorder expansion. The example presents a good opportunity to demonstrate application of these methods but this is not taken far beyond calculation of the diagnostics.
I see two possibilities for the paper:
1) Repackage it as a technical note focusing on the calculation of the diagnostics
2) Extend the work to generate diagnostics of overall inversion performance, probably focusing on robustness.Specific Comments:
L130: What is a footprintinduced probability distribution?
L190: The definition of the averaging kernel is true but this is an odd motivation for it, much better below when contemplating sensitivity of result to prior
Eq. 19: it's worth noting that this sensitivity is very close to the proportional uncertainty reduction A P^1 and hence the averaging kernel. By the way I thank the authors for making me think hard enough about the relationship between AK, DOFS and uncertainty reduction to finally get an intuitive sense of it
L275: When commenting on covariances between H, Q etc we should also note that constraints like conservation of mass introduce strong covariances within the parameters of H. Covariances can only occur on physically plausible manifolds. This is a profoundly understudied problem in transport modelling and there is probably great insight to be borrowed from Numerical Weather Prediction.
L431: not sure what the authors mean by aggregation error here. If they're truly commenting on temporal aggregation error they should cite DOI:10.5194/acp1134432011.Citation: https://doi.org/10.5194/gmd202289RC1 
AC1: 'Reply on RC1', Vineet Yadav, 07 Sep 2022
The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd202289/gmd202289AC1supplement.pdf

AC1: 'Reply on RC1', Vineet Yadav, 07 Sep 2022

RC2: 'Comment on gmd202289', Anonymous Referee #2, 15 Jul 2022
This paper presents methods for sensitivity analysis of the solution of the inverse problem to various input parameters, keeping in mind the cost associated with the sensitivity computations. This is important because it provides the user with some understanding of the sensitivities and may lead to better prior models or measurement strategies. The methodology is illustrated in the case of an atmospheric inverse problem. The computational costs are significant for such problems, even for computing the MAP estimate, so it is important to derive efficient methods for sensitivity analysis.
Major concerns:
 Contributions: I couldn’t fully understand the contributions of this paper. Is it claiming to be developing new methods for inverse problems, or applying existing techniques to the case study? The methods for local sensitivity analysis involving derivatives appear to be a special case of the Hyperdifferential sensitivity analysis for linear inverse problems and the GSA method is essentially DGSM (see end of review).
 Mathematical exposition: I understand that this is not an applied math journal, but the standard for exposition was well below this journal. The notation is not setup properly and inconsistently used. There were a lot of vague statements (I did not fully tabulate this list).
Specific comments:
 Title: I think working in the word sensitivity is better here than “assessing”.
 The abstract does not clarify, of what quantity (i.e. MAP estimate) the sensitivity is being computed. GHG is not expanded and Jacobian should be capitalized throughout. The word Jacobian (to me) is misleading because this is a linear problem – there are other Jacobians used in the paper.
 The introduction does not clearly list the contributions (as mentioned earlier) and is missing some references.
 Section 3:
 The inversion is not setup properly before 3.1.13.1.3 are explained. I essentially did not understand anything in these subsections. I am not sure what sets are being considered, how this relates to the inversion, etc. There is a small discussion at the end of 3.1.3 but it is referring to things that haven’t yet been defined.
 Section 3.2: Someone not familiar with this material will struggle since it has not been discussed properly. I suggest reorganizing this section bringing some of this material earlier.
 Notation/Writing: This is not consistent throughout the paper. Sometimes subscripts refer to sizes, other times they mean elements. Sometimes boldfaced, sometimes not. Sometimes lower/upper case. Sentences should not start with variables. When units are being described, the entries of vectors have units, not the vectors themselves.
 Line 157: it’s > its. This occurred in other places also.
 Line 169: it can be simplified using the notation below in 172.
 Line 173: One differentiates a function rather than an equation.
 When referring to equations, one typically puts a parenthesis, e.g. (10).
 Line 178: The inputs to the inverse problems are data and hyperparameters. The transport model is typically fixed, as is the drift matrix X. What does it mean to differentiate wrt them? Why is that useful in applications? Are these matrices functions of some hyperparameters?
 Line 185: The equation is not referred to as Lambda but one of the terms there is Lambda.
 Line 203: When differentiating a vector wrt a matrix, one should get a tensor. I think what is happening is that you are vectorizing X and then differentiating. But the notation is not clear. Same with line 227.
 Section 3.3: I agree that full GSA is very complicated. It’s not clear how this is GSA. Is this essentially derivative based global sensitivity? Once again I did not really understand what was being discussed.
 Section 3.4: The authors raise a good point here cautioning against sensitivity metrics since they are of different units but little has been done to address them. There are techniques in sensitivity analysis that the authors should consult (DGSMs and activity scores are some techniques, see Constantine and Diaz)
 Section 4: I didn’t understand the figures since sometimes the sizes of the quantities are not clear. Are entrywise sensitivities being plotted?
 Appendix: The notation was especially problematic here.
Returning to the major concerns: I don’t think the paper is novel in the methodology, but as an application to atmospheric inverse problems I think it has potential. In its current version, I don’t think it is worthy of publishing. In addition to the comments I raised, I would encourage the authors to think of the following questions to improve the utility. What is the computational cost of these approaches? Can they be computed when the forward model is not available entrywise? What are the challenges involved and how can they be efficiently implemented? What are the strengths and weaknesses of each approach? How does one compare sensitivities of different quantities (with different units) and rank the sources of sensitivity?
References:
 Sunseri, Isaac, et al. "Hyperdifferential sensitivity analysis for inverse problems constrained by partial differential equations." Inverse Problems 36.12 (2020): 125001.
 Sunseri, Isaac, et al. "Hyperdifferential sensitivity analysis for nonlinear Bayesian inverse problems." arXiv preprint arXiv:2202.02219 (2022).
 Sobol, I. M., and S. Kucherenko. "Derivative based global sensitivity measures." ProcediaSocial and Behavioral Sciences 2.6 (2010): 77457746.
 Constantine, Paul G., and Paul Diaz. "Global sensitivity metrics from active subspaces." Reliability Engineering & System Safety 162 (2017): 113.
Citation: https://doi.org/10.5194/gmd202289RC2 
AC2: 'Reply on RC2', Vineet Yadav, 07 Sep 2022
The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd202289/gmd202289AC2supplement.pdf
Vineet Yadav et al.
Vineet Yadav et al.
Viewed
HTML  XML  Total  Supplement  BibTeX  EndNote  

800  162  17  979  36  2  2 
 HTML: 800
 PDF: 162
 XML: 17
 Total: 979
 Supplement: 36
 BibTeX: 2
 EndNote: 2
Viewed (geographical distribution)
Country  #  Views  % 

Total:  0 
HTML:  0 
PDF:  0 
XML:  0 
 1