Submitted as: methods for assessment of models 27 Sep 2021
Submitted as: methods for assessment of models  27 Sep 2021
Earth System Model Parameter Adjustment Using a Green's Functions Approach
 ^{1}Agricultural Research Organization, Rishon Lezion, Israel
 ^{2}Goddard Space Flight Center, Greenbelt, MD, US
 ^{3}Science Systems and Applications Inc., Greenbelt, MD, US
 ^{4}Massachusetts Institute of Technology, Cambridge, MA, US
 ^{5}Jet Propulsion Laboratory,California Institute of Technology, Pasadena, CA, US
 ^{1}Agricultural Research Organization, Rishon Lezion, Israel
 ^{2}Goddard Space Flight Center, Greenbelt, MD, US
 ^{3}Science Systems and Applications Inc., Greenbelt, MD, US
 ^{4}Massachusetts Institute of Technology, Cambridge, MA, US
 ^{5}Jet Propulsion Laboratory,California Institute of Technology, Pasadena, CA, US
Abstract. We demonstrate the practicality and effectiveness of using a Green's functions estimation approach for adjusting uncertain parameters in an Earth System Model (ESM). This estimation approach had previously been applied to an intermediatecomplexity climate model and to individual ESM components, e.g., ocean, seaice, or carboncycle components. Here, the Green's functions approach is applied to a stateoftheart ESM that comprises a global atmosphereland configuration of the Goddard Earth Observing System (GEOS) coupled to an ocean and seaice configuration of the Massachusetts Institute of Technology general circulation model (MITgcm). Horizontal grid spacing is approximately 110 km for GEOS and 37–110 km for MITgcm. In addition to the reference GEOSMITgcm simulation, we carry out a series of model sensitivity experiments, in which 20 uncertain parameters are perturbed. These control
parameters can be used to adjust seaice, microphysics, turbulence, radiation, and surface schemes in the coupled simulation. We define eight observational targets: seaice fraction, net surface shortwave radiation, downward longwave radiation, nearsurface temperature, sea surface temperature, sea surface salinity, and ocean temperature and salinity at 300 m. We applied the Green's functions approach to optimize the values of the 20 control parameters so as to minimize a weighted leastsquares distance between the model and the eight observational targets. The new experiment with the optimized parameters resulted in a total cost reduction of 9 % relative to a simulation that had already been adjusted using other methods. The optimized experiment attained a balanced cost reduction over most of the observational targets. We also report on results from a set of sensitivity experiments that are not used in the final optimized simulation but helped explore options and guided the optimization process. These experiments include an assessment of sensitivity to the number of control parameters and to the selection of observational targets and weights in the cost function. Based on these sensitivity experiments, we selected a specific definition for the cost function. The sensitivity experiments also revealed a decreasing overall cost as the number of control variables was increased. In summary, we recommend using the Green's functions estimation approach as an additional finetuning step in the model development process. The method is not a replacement for modelers' experience in choosing and adjusting sensitive model parameters. Instead, it is an additional practical and effective tool for carrying out final adjustments of uncertain ESM parameters.
Ehud Strobach et al.
Status: final response (author comments only)

RC1: 'Comment on gmd2021251', Anonymous Referee #1, 02 Nov 2021
This papers proposes an objective method to tune ESMs in a global approach. The authors proposes a mathematical equivalent to the way modellers tune their models so as to rationalize and automatize the tuning process, based on a Green’s functions approach. This is a very important topic for ESM developers that come to a limitation in the model developments due to the long process of humanmade iterative tuning. This study is perfectly timely. The authors provides an example of application and discuss some of the assumptions made. However, some assumptions are not fully explained or would deserve more comments. I therefore recommend the manuscript to be accepted after minor revisions.
Main comments
Section 3: Explaining the methodology using a mathematical formalism is a very good idea, but in the present form, I have not the impression that it is sufficiently clear. Maybe I am not a good mathematician but I think that this manuscript should be well understood by modellers and improving the explanation would leverage the impact of the study. Moreover, the equations provided in the manuscript are already presented in Menemenlis et al. (2005), therefore, they do not provide anew outlook. I would suggest to better explain these equations here. It is not clear to me how equation (4) is established? Similarly, it is not clear to me what is the difference between G(Æ) and G Æ? Maybe the notations could be simplified. Additionally, the physical meaning of Q and R should be detailed as well as the methodology to set them. For instance in section 4.3, it is said that “R is chosen diagonal”: apart from being mathematically convenient, what are the physical hypothesis behind such a choice? What makes this choice sensible? Isn’t it a strong hypothesis? What happens if reality is far from diagonal? These points should be discussed with a physical point of view in the manuscript to better illustrate the advantages and flaws of the method used.
Similarly, section 5.1 is a bit hard to read. The physical idea behind the choice of functions is not clear enough. What is the physical meaning of Q choices proposed? This section need to be detailed to better catch the physical hypothesis that are made depending on each choice of covariance matrix.
There is also a crucial point that should be at least discussed in the paper: in such complex ESM there is a long spinup process of several decades that has to be taken in to account. If only taking the first 10 years of ESM simulations for tuning, what is the result of such a sort term tuning on the long term equilibrium? How do the authors proposes to handle this problem?
Line 230: the fact that the surface temperature is less well reproduced after the tuning process is very interesting. Even if it reflects a flaw in the present choice of observational targets, the discussion on this should be enriched. What type of other observational targets could be proposed to avoid this?
Section 5.2: by increasing the number of parameters, there is a risk to overfit the model to the set of observation chosen as discussed in Dommenget and Rezny, (2018). This risk should be discussed in the last section. Additionally, by increasing the number of parameters the final fit is necessary improved as more degree of freedom have been used. Do the authors could use a methods similar to the LASSO technique (Tibshirani, 1996, used in linear regression) to penalize the results according to the number of degree of freedom? This would help selecting only the more useful parameters and avoid overfitting.
Ln 304305: ”The largest cost reduction was found between 5 to 10 optimized parameters.” And what about the ratio projected cost/optimized cost? Maybe this information could be added on figure 6. Is this ratio stable depending on the number of parameters?
Section 5.4: what would be the cost if the first 5 years were used? Would it be different from the last five years? If yes, this would probably says that the spinup is important. But then, what about taking the last 10 years over a 20 years simulation. The initial choice of 10 years appears as a bit short against the spinup timescale of coupled models. The spinup problem would be discussed more thoroughly. Maybe it would be nice to show the model initial drift.
I may have misunderstood the methodology, but reading again the introduction section, I am not convinced that the proposed methodology addresses all the points raised in the introduction.
 ln 4952 :“An experiment with both perturbed would not reveal which had the greater impact on the solution, and experiments with only one at a time perturbed would not behave as an experiment with both perturbed. The sequential methodology, therefore, for this example, may result in a suboptimal combination of these parameters.” I do not understand how the Green’s function approach do something different to “one at a time” perturbed experiment. Maybe it is just a question of formulation here?
 ln 58 :”The second drawback is the large number of needed simulations, since a new set of experiments is required for each set of observational targets”. Using the standard by hand method, developers usually have a look at a panel of diagnostics and do not need to redo the experiments for a new diagnostic, I do not see any difference on this point. This point is repeated on line 74, I would not emphasize this much.
 ln 7171: “The interdependence is accounted for in the constraints on the error covariance for each parameter.” This points of interdependence is not well developed in the manuscript. I would promote to better illustrate this point.
Miscellaneous
Section 2: please indicate the ocean model timestep and better separate the ocean and atmosphere models description in two distinct paragraphs.
Table 4 and 5: these tables are large and difficult to interpret as is, the exact values not behind of strong interest. Please consider replacing them by plots.
Section 5.2: Concerning LW and SW radiation, is the change from downward to net? Why changing this point? Could the authors discuss a bit more this choice?
References :
Dommenget, D. and Rezny, M.: A Caveat Note on Tuning in the Development of Coupled Climate Models, J. Adv. Model. Earth Syst., 10, 78–97, https://doi.org/10.1002/2017MS000947, 2018.
Tibshirani, R.: Regression Shrinkage and Selection Via the Lasso, J. R. Stat. Soc. Ser. B Methodol., 58, 267–288, https://doi.org/10.1111/j.25176161.1996.tb02080.x, 1996.

RC2: 'Comment on gmd2021251', Anonymous Referee #2, 12 Dec 2021
In this work the authors used Green's function approach to optimize parameters in an earth system model. This optimization was achieved by minimizing a weighted leastsquares distance. The numerical results demonstrated that the Green's functions estiamtion approach can be a good finetuning step in the model development process to adjust uncertain parameters of the model considered in this work. Also, sensitvity analysis results were presented to further investigate the effectiveness of the Greens's function approach. In general, as an appliction work, the manuscript is well organized and technical details are clear.
I have a few comments on the approach itself although this work is focused on application.
1. Eq(5) is the combination of Eq (1) and Eq (3), and G seems like a composite of M_i and H. How do we understand it as an "convolution of observation operator H and M_i"?
2. Eq(4) and Eq (9) are similar to some formulas in Kalman filter. Indeed, Eqs (2) and (3) are also the starting point of Kalman filter. What is the main difference between the Green's function method and Kalman filter? Or are they equivalent to each other to some extent or in some cases?
3. The impact of choosing different Q was discussed using Q=\infty and Q=I. How about the influence of R? Here R is a diagonal matrix, which indicates that the errors for different observables are independent (which is a quite standard setup in Kalman filter). How about the errors are not independent? Any comments or intuitions?
Ehud Strobach et al.
Ehud Strobach et al.
Viewed
HTML  XML  Total  BibTeX  EndNote  

522  83  12  617  6  3 
 HTML: 522
 PDF: 83
 XML: 12
 Total: 617
 BibTeX: 6
 EndNote: 3
Viewed (geographical distribution)
Country  #  Views  % 

Total:  0 
HTML:  0 
PDF:  0 
XML:  0 
 1