Metrics for assessing Linear Inverse Problems: a case study of a Trace Gas Inversion

Yadav, Vineet; Ghosh, Subhomoy; Miller, Charles E.

doi:https://doi.org/10.5194/gmd-2022-89

Preprints

https://doi.org/10.5194/gmd-2022-89

Preprints

Submitted as: methods for assessment of models

23 May 2022

Submitted as: methods for assessment of models |

| 23 May 2022

Status: a revised version of this preprint was accepted for the journal GMD and is expected to appear here in due course.

Metrics for assessing Linear Inverse Problems: a case study of a Trace Gas Inversion

Vineet Yadav, Subhomoy Ghosh, and Charles E. Miller

Abstract. Multiple metrics have been proposed and utilized to assess the performance of linear Bayesian and geostatistical inverse problems. These metrics are mostly related to assessing reduction in prior uncertainties, comparing modeled observations to true observations, and checking distributional assumptions. These metrics though important should be augmented with sensitivity analysis to obtain a comprehensive understanding of the performance of inversions and critically improve confidence in the estimated fluxes. With this motivation, we derive analytical forms of the local sensitivities with respect to the number of inputs such as measurements, covariance parameters, covariates, and forward operator or jacobian. In addition to local sensitivity, we develop a framework for global sensitivity analysis that shows the apportionment of the uncertainty of different inputs to an inverse problem. The proposed framework is applicable to any other domain that employs linear Bayesian and geostatistical inverse methods. We show the application of our methodology in the context of an atmospheric inverse problem for estimating urban GHG emissions in Los Angeles. Within its context, we also propose a mathematical framework to construct correlation functions and components of uncertainty matrices from a pre-computed jacobian that encompasses non-stationary structures.

Received: 26 Mar 2022 – Discussion started: 23 May 2022

Download & links

Preprint (PDF, 8350 KB)

Supplement (6269 KB)

Vineet Yadav et al.

Status: closed

RC1:
'Comment on gmd-2022-89', Peter Rayner, 03 Jul 2022

This paper presents a series of quantities that can be derived from linear inverse theory. Put roughly they are the similarity of footprints (nearly the independence of rows of the Jacobian), the local sensitivity of the result to various inputs and finally a global sensitivity using a first-order Taylor expansion with respect to all inputs. The metrics are potentially useful and some are, to my knowledge, novel. The paper is potentially in scope for AMT though I think it needs more work to make it more relevant to likely readers.

My first problem with the paper is its title. the word "assessment" suggests some comment on the quality or robustness of an inversion. The authors don't do that and it's not clear from the paper that the generated metrics can do it. I'm unclear, for example, what new information is provided by the overlap of footprints. It might well mean that parts of the control space are under-sampled by the observations but the posterior uncertainty already tells us this. For the linear Gaussian case the posterior uncertainty can be calculated without measurements.

Likewise the sensitivity of the posterior estimate to the value of a given measurement is potentially useful as a warning flag for measurements that might have undue control on the outcome but it's not really developed. the global sensitivity analysis, which allows consideration of all inputs to the linear inverse problem, is potentially more interesting but again is not developed beyond generation of the first-order expansion. The example presents a good opportunity to demonstrate application of these methods but this is not taken far beyond calculation of the diagnostics.

I see two possibilities for the paper:

1) Repackage it as a technical note focusing on the calculation of the diagnostics

2) Extend the work to generate diagnostics of overall inversion performance, probably focusing on robustness.

Specific Comments:

L130: What is a footprint-induced probability distribution?

L190: The definition of the averaging kernel is true but this is an odd motivation for it, much better below when contemplating sensitivity of result to prior

Eq. 19: it's worth noting that this sensitivity is very close to the proportional uncertainty reduction A P^-1 and hence the averaging kernel. By the way I thank the authors for making me think hard enough about the relationship between AK, DOFS and uncertainty reduction to finally get an intuitive sense of it

L275: When commenting on covariances between H, Q etc we should also note that constraints like conservation of mass introduce strong covariances within the parameters of H. Covariances can only occur on physically plausible manifolds. This is a profoundly under-studied problem in transport modelling and there is probably great insight to be borrowed from Numerical Weather Prediction.

L431: not sure what the authors mean by aggregation error here. If they're truly commenting on temporal aggregation error they should cite DOI:10.5194/acp-11-3443-2011.

Citation: https://doi.org/10.5194/gmd-2022-89-RC1
- AC1: 'Reply on RC1', Vineet Yadav, 07 Sep 2022
  
  The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2022-89/gmd-2022-89-AC1-supplement.pdf
  
  Citation: https://doi.org/10.5194/gmd-2022-89-AC1
RC2:
'Comment on gmd-2022-89', Anonymous Referee #2, 15 Jul 2022
This paper presents methods for sensitivity analysis of the solution of the inverse problem to various input parameters, keeping in mind the cost associated with the sensitivity computations. This is important because it provides the user with some understanding of the sensitivities and may lead to better prior models or measurement strategies. The methodology is illustrated in the case of an atmospheric inverse problem. The computational costs are significant for such problems, even for computing the MAP estimate, so it is important to derive efficient methods for sensitivity analysis.

Major concerns:

Contributions: I couldn’t fully understand the contributions of this paper. Is it claiming to be developing new methods for inverse problems, or applying existing techniques to the case study? The methods for local sensitivity analysis involving derivatives appear to be a special case of the Hyperdifferential sensitivity analysis for linear inverse problems and the GSA method is essentially DGSM (see end of review).

Mathematical exposition: I understand that this is not an applied math journal, but the standard for exposition was well below this journal. The notation is not setup properly and inconsistently used. There were a lot of vague statements (I did not fully tabulate this list).

Specific comments:

Title: I think working in the word sensitivity is better here than “assessing”.

The abstract does not clarify, of what quantity (i.e. MAP estimate) the sensitivity is being computed. GHG is not expanded and Jacobian should be capitalized throughout. The word Jacobian (to me) is misleading because this is a linear problem – there are other Jacobians used in the paper.

The introduction does not clearly list the contributions (as mentioned earlier) and is missing some references.

Section 3:

The inversion is not setup properly before 3.1.1-3.1.3 are explained. I essentially did not understand anything in these subsections. I am not sure what sets are being considered, how this relates to the inversion, etc. There is a small discussion at the end of 3.1.3 but it is referring to things that haven’t yet been defined.

Section 3.2: Someone not familiar with this material will struggle since it has not been discussed properly. I suggest reorganizing this section bringing some of this material earlier.

Notation/Writing: This is not consistent throughout the paper. Sometimes subscripts refer to sizes, other times they mean elements. Sometimes boldfaced, sometimes not. Sometimes lower/upper case. Sentences should not start with variables. When units are being described, the entries of vectors have units, not the vectors themselves.

Line 157: it’s -> its. This occurred in other places also.

Line 169: it can be simplified using the notation below in 172.

Line 173: One differentiates a function rather than an equation.

When referring to equations, one typically puts a parenthesis, e.g. (10).

Line 178: The inputs to the inverse problems are data and hyperparameters. The transport model is typically fixed, as is the drift matrix X. What does it mean to differentiate wrt them? Why is that useful in applications? Are these matrices functions of some hyperparameters?

Line 185: The equation is not referred to as Lambda but one of the terms there is Lambda.

Line 203: When differentiating a vector wrt a matrix, one should get a tensor. I think what is happening is that you are vectorizing X and then differentiating. But the notation is not clear. Same with line 227.

Section 3.3: I agree that full GSA is very complicated. It’s not clear how this is GSA. Is this essentially derivative based global sensitivity? Once again I did not really understand what was being discussed.

Section 3.4: The authors raise a good point here cautioning against sensitivity metrics since they are of different units but little has been done to address them. There are techniques in sensitivity analysis that the authors should consult (DGSMs and activity scores are some techniques, see Constantine and Diaz)

Section 4: I didn’t understand the figures since sometimes the sizes of the quantities are not clear. Are entrywise sensitivities being plotted?

Appendix: The notation was especially problematic here.

Returning to the major concerns: I don’t think the paper is novel in the methodology, but as an application to atmospheric inverse problems I think it has potential. In its current version, I don’t think it is worthy of publishing. In addition to the comments I raised, I would encourage the authors to think of the following questions to improve the utility. What is the computational cost of these approaches? Can they be computed when the forward model is not available entrywise? What are the challenges involved and how can they be efficiently implemented? What are the strengths and weaknesses of each approach? How does one compare sensitivities of different quantities (with different units) and rank the sources of sensitivity?

References:

Sunseri, Isaac, et al. "Hyper-differential sensitivity analysis for inverse problems constrained by partial differential equations." 36.12 (2020): 125001.

Sunseri, Isaac, et al. "Hyper-differential sensitivity analysis for nonlinear Bayesian inverse problems." (2022).

Sobol, I. M., and S. Kucherenko. "Derivative based global sensitivity measures." 2.6 (2010): 7745-7746.

Constantine, Paul G., and Paul Diaz. "Global sensitivity metrics from active subspaces." 162 (2017): 1-13.
Citation: https://doi.org/10.5194/gmd-2022-89-RC2
- AC2: 'Reply on RC2', Vineet Yadav, 07 Sep 2022
  
  The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2022-89/gmd-2022-89-AC2-supplement.pdf
  
  Citation: https://doi.org/10.5194/gmd-2022-89-AC2

Status: closed

RC1:
'Comment on gmd-2022-89', Peter Rayner, 03 Jul 2022

This paper presents a series of quantities that can be derived from linear inverse theory. Put roughly they are the similarity of footprints (nearly the independence of rows of the Jacobian), the local sensitivity of the result to various inputs and finally a global sensitivity using a first-order Taylor expansion with respect to all inputs. The metrics are potentially useful and some are, to my knowledge, novel. The paper is potentially in scope for AMT though I think it needs more work to make it more relevant to likely readers.

My first problem with the paper is its title. the word "assessment" suggests some comment on the quality or robustness of an inversion. The authors don't do that and it's not clear from the paper that the generated metrics can do it. I'm unclear, for example, what new information is provided by the overlap of footprints. It might well mean that parts of the control space are under-sampled by the observations but the posterior uncertainty already tells us this. For the linear Gaussian case the posterior uncertainty can be calculated without measurements.

Likewise the sensitivity of the posterior estimate to the value of a given measurement is potentially useful as a warning flag for measurements that might have undue control on the outcome but it's not really developed. the global sensitivity analysis, which allows consideration of all inputs to the linear inverse problem, is potentially more interesting but again is not developed beyond generation of the first-order expansion. The example presents a good opportunity to demonstrate application of these methods but this is not taken far beyond calculation of the diagnostics.

I see two possibilities for the paper:

1) Repackage it as a technical note focusing on the calculation of the diagnostics

2) Extend the work to generate diagnostics of overall inversion performance, probably focusing on robustness.

Specific Comments:

L130: What is a footprint-induced probability distribution?

L190: The definition of the averaging kernel is true but this is an odd motivation for it, much better below when contemplating sensitivity of result to prior

Eq. 19: it's worth noting that this sensitivity is very close to the proportional uncertainty reduction A P^-1 and hence the averaging kernel. By the way I thank the authors for making me think hard enough about the relationship between AK, DOFS and uncertainty reduction to finally get an intuitive sense of it

L275: When commenting on covariances between H, Q etc we should also note that constraints like conservation of mass introduce strong covariances within the parameters of H. Covariances can only occur on physically plausible manifolds. This is a profoundly under-studied problem in transport modelling and there is probably great insight to be borrowed from Numerical Weather Prediction.

L431: not sure what the authors mean by aggregation error here. If they're truly commenting on temporal aggregation error they should cite DOI:10.5194/acp-11-3443-2011.

Citation: https://doi.org/10.5194/gmd-2022-89-RC1
- AC1: 'Reply on RC1', Vineet Yadav, 07 Sep 2022
  
  The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2022-89/gmd-2022-89-AC1-supplement.pdf
  
  Citation: https://doi.org/10.5194/gmd-2022-89-AC1
RC2:
'Comment on gmd-2022-89', Anonymous Referee #2, 15 Jul 2022
This paper presents methods for sensitivity analysis of the solution of the inverse problem to various input parameters, keeping in mind the cost associated with the sensitivity computations. This is important because it provides the user with some understanding of the sensitivities and may lead to better prior models or measurement strategies. The methodology is illustrated in the case of an atmospheric inverse problem. The computational costs are significant for such problems, even for computing the MAP estimate, so it is important to derive efficient methods for sensitivity analysis.

Major concerns:

Contributions: I couldn’t fully understand the contributions of this paper. Is it claiming to be developing new methods for inverse problems, or applying existing techniques to the case study? The methods for local sensitivity analysis involving derivatives appear to be a special case of the Hyperdifferential sensitivity analysis for linear inverse problems and the GSA method is essentially DGSM (see end of review).

Mathematical exposition: I understand that this is not an applied math journal, but the standard for exposition was well below this journal. The notation is not setup properly and inconsistently used. There were a lot of vague statements (I did not fully tabulate this list).

Specific comments:

Title: I think working in the word sensitivity is better here than “assessing”.

The abstract does not clarify, of what quantity (i.e. MAP estimate) the sensitivity is being computed. GHG is not expanded and Jacobian should be capitalized throughout. The word Jacobian (to me) is misleading because this is a linear problem – there are other Jacobians used in the paper.

The introduction does not clearly list the contributions (as mentioned earlier) and is missing some references.

Section 3:

The inversion is not setup properly before 3.1.1-3.1.3 are explained. I essentially did not understand anything in these subsections. I am not sure what sets are being considered, how this relates to the inversion, etc. There is a small discussion at the end of 3.1.3 but it is referring to things that haven’t yet been defined.

Section 3.2: Someone not familiar with this material will struggle since it has not been discussed properly. I suggest reorganizing this section bringing some of this material earlier.

Notation/Writing: This is not consistent throughout the paper. Sometimes subscripts refer to sizes, other times they mean elements. Sometimes boldfaced, sometimes not. Sometimes lower/upper case. Sentences should not start with variables. When units are being described, the entries of vectors have units, not the vectors themselves.

Line 157: it’s -> its. This occurred in other places also.

Line 169: it can be simplified using the notation below in 172.

Line 173: One differentiates a function rather than an equation.

When referring to equations, one typically puts a parenthesis, e.g. (10).

Line 178: The inputs to the inverse problems are data and hyperparameters. The transport model is typically fixed, as is the drift matrix X. What does it mean to differentiate wrt them? Why is that useful in applications? Are these matrices functions of some hyperparameters?

Line 185: The equation is not referred to as Lambda but one of the terms there is Lambda.

Line 203: When differentiating a vector wrt a matrix, one should get a tensor. I think what is happening is that you are vectorizing X and then differentiating. But the notation is not clear. Same with line 227.

Section 3.3: I agree that full GSA is very complicated. It’s not clear how this is GSA. Is this essentially derivative based global sensitivity? Once again I did not really understand what was being discussed.

Section 3.4: The authors raise a good point here cautioning against sensitivity metrics since they are of different units but little has been done to address them. There are techniques in sensitivity analysis that the authors should consult (DGSMs and activity scores are some techniques, see Constantine and Diaz)

Section 4: I didn’t understand the figures since sometimes the sizes of the quantities are not clear. Are entrywise sensitivities being plotted?

Appendix: The notation was especially problematic here.

Returning to the major concerns: I don’t think the paper is novel in the methodology, but as an application to atmospheric inverse problems I think it has potential. In its current version, I don’t think it is worthy of publishing. In addition to the comments I raised, I would encourage the authors to think of the following questions to improve the utility. What is the computational cost of these approaches? Can they be computed when the forward model is not available entrywise? What are the challenges involved and how can they be efficiently implemented? What are the strengths and weaknesses of each approach? How does one compare sensitivities of different quantities (with different units) and rank the sources of sensitivity?

References:

Sunseri, Isaac, et al. "Hyper-differential sensitivity analysis for inverse problems constrained by partial differential equations." 36.12 (2020): 125001.

Sunseri, Isaac, et al. "Hyper-differential sensitivity analysis for nonlinear Bayesian inverse problems." (2022).

Sobol, I. M., and S. Kucherenko. "Derivative based global sensitivity measures." 2.6 (2010): 7745-7746.

Constantine, Paul G., and Paul Diaz. "Global sensitivity metrics from active subspaces." 162 (2017): 1-13.
Citation: https://doi.org/10.5194/gmd-2022-89-RC2
- AC2: 'Reply on RC2', Vineet Yadav, 07 Sep 2022
  
  The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2022-89/gmd-2022-89-AC2-supplement.pdf
  
  Citation: https://doi.org/10.5194/gmd-2022-89-AC2

Vineet Yadav et al.

Supplement

https://doi.org/10.5194/gmd-2022-89-supplement

Vineet Yadav et al.

Viewed

Total article views: 979 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	Supplement	BibTeX	EndNote
800	162	17	979	36	2	2

HTML: 800
PDF: 162
XML: 17
Total: 979
Supplement: 36
BibTeX: 2
EndNote: 2

Views and downloads (calculated since 23 May 2022)

Month	HTML	PDF	XML	Total
May 2022	165	42	2	209
Jun 2022	99	18	1	118
Jul 2022	103	16	6	125
Aug 2022	47	12	1	60
Sep 2022	80	17	4	101
Oct 2022	42	5	0	47
Nov 2022	46	6	1	53
Dec 2022	31	12	0	43
Jan 2023	32	4	0	36
Feb 2023	54	12	1	67
Mar 2023	47	6	0	53
Apr 2023	47	9	0	56
May 2023	7	3	1	11

Cumulative views and downloads (calculated since 23 May 2022)

Month	HTML	PDF	XML	Total
May 2022	165	42	2	209
Jun 2022	99	18	1	118
Jul 2022	103	16	6	125
Aug 2022	47	12	1	60
Sep 2022	80	17	4	101
Oct 2022	42	5	0	47
Nov 2022	46	6	1	53
Dec 2022	31	12	0	43
Jan 2023	32	4	0	36
Feb 2023	54	12	1	67
Mar 2023	47	6	0	53
Apr 2023	47	9	0	56
May 2023	7	3	1	11

Viewed (geographical distribution)

Total article views: 915 (including HTML, PDF, and XML) Thereof 915 with geography defined and 0 with unknown origin.

Country	#	Views	%

Latest update: 01 Jun 2023

Short summary

Measuring the performance of inversions in linear Bayesian problems is crucial in real-life applications. In this work, we provide analytical forms of the local and global sensitivities of the estimated fluxes with respect to various inputs. We provide methods to uniquely map the observational signal to spatio-temporal domains. Utilizing this, we also show techniques to assess correlations between the jacobians that naturally translate to nonstationary covariance matrix components.


Total:	0
HTML:	0
PDF:	0
XML:	0