Sensitivity analysis of a data-driven model of ocean temperature

Furner, Rachel; Haynes, Peter; Munday, Dave; Paige, Brooks; Jones, Daniel C.; Shuckburgh, Emily

doi:https://doi.org/10.5194/gmd-2021-132

Preprints

https://doi.org/10.5194/gmd-2021-132

Preprints

Submitted as: model description paper

02 Jun 2021

Submitted as: model description paper |

| 02 Jun 2021

Status: this preprint was under review for the journal GMD but the revision was not accepted.

Sensitivity analysis of a data-driven model of ocean temperature

Rachel Furner, Peter Haynes, Dave Munday, Brooks Paige, Daniel C. Jones, and Emily Shuckburgh

Abstract. There has been much recent interest in developing data-driven models for weather and climate predictions. These have shown reasonable success in modelling atmospheric dynamics over short time scales, however there are open questions regarding the sensitivity and robustness of these models. Using model interpretation techniques to better understand how data-driven models are making predictions is critical to developing trust in these alternative prediction systems. We develop a simple regression model of ocean temperature evolution, Ocean Temperature Regressor v1.0, and investigate its sensitivity to improve understanding of whether data-driven models are capable of learning the complex underlying dynamics of the systems being modelled.

We investigate model sensitivity in a variety of ways and find that Ocean Temperature Regressor v1.0 behaves in ways which are, for the most part, in line with our knowledge of the ocean system being modelled. Specifically we see that the regressor heavily bases its forecasts on, and is dependent on, variables which we know are key to the physical dynamics inherent in the system, such as the currents and density. By contrast, inputs to the regressor which have limited direct dynamic impact, such as location, are not heavily used by the regressor. We also find that the regression model requires non-linear interactions between inputs in order to show any meaningful predictive skill – in line with out knowledge of the highly nonlinear dynamics of the ocean. Further sensitivity analysis is carried out to interpret that the ways in which certain variables are used by the regression model. Results here are again mostly in line with our physical knowledge of the system, for example, we see that information about the vertical profile of the water column reduces errors in areas associated with convective activity, and information about the currents is used by the regressor to reduce errors in regions dominated by advective processes.

Our results show that even a simple regression model is capable of “learning” much of the physical dynamics inherent in the ocean system being modelled, which gives promise for the sensitivity and generalisability of data-driven models more generally.

Received: 21 Apr 2021 – Discussion started: 02 Jun 2021

Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this preprint. The responsibility to include appropriate place names lies with the authors.

Download & links

Rachel Furner, Peter Haynes, Dave Munday, Brooks Paige, Daniel C. Jones, and Emily Shuckburgh

Status: closed

CC1:
'Comment on gmd-2021-132', Paul PUKITE, 06 Jun 2021
"Specifically, we analyse the coefficients of our regression model and find that the predictions for a grid cell are based

heavily on the density at the surrounding points, and the interaction between temperature at the grid cell and its neighbouring

points. The importance of temperature interaction at surrounding points is representative of advective and diffusive processes

which take place across the domain. The importance of density is in line with the the simulator representing, to some extent,

density driven currents which are responsible for much of the changes in temperature in this GCM configuration"

Density drives climate variability in terms of El Nino/La Nina cycles via the bouyancy of the themocline, where slight density changes above and below the thermocline will impact the effective gravity at the interface. How much of the regression model is sensitive to this narrow equatorial region? Or is it the larger AMO overturning circulation that is a stronger regresssor?
Citation: https://doi.org/10.5194/gmd-2021-132-CC1
- AC1: 'Reply on CC1', Rachel Furner, 21 Jun 2021
  
  Thanks for your comment and the interest in our paper.
  In this instance we’ve intentionally chosen a very idealised and simplified set up, allowing us to more easily analyse the behaviour of the regressor in relation to the simulator. This means the underlying MITgcm simulation which the regressor is learning has constant surface forcing, so the training dataset has no El Nino/La Nina cycle, and hence the regressor (which is intended to mimic this dataset) also won’t show any signal related to El Nino/La Nina.
  It is however an interesting question as to how much the regressor would be able to learn from a more realistic set up which did include El Nino/La Nina, given the relatively small geographical extent of this. In our set up the regressor is learning a single equation, to be applied at each grid box in the domain (rather than learning specifics about the dynamics of different regions) and it is learning the (relatively) small scale, local dynamics, rather than large scale patterns such as El Nino/La Nina. In this sense it is very similar to the way that traditional GCMs make their predictions – values for a variable at a grid box are changed according to a consistent algorithm which takes as inputs the values at the grid cell and its surrounding cells. This grid-cell scale physics then leads to the wider circulation features such as El Nino/La Nina rather than these being incorporated into the GCM directly.
  Given this we might reasonably expect the regressor to capture the cell-level dynamics which lead to larger features such as El Nino/La Nina in the same way as GCMs do. This is however dependent on how closely the regressor learns the physics of the system.
  
  Citation: https://doi.org/10.5194/gmd-2021-132-AC1
- AC2: 'Reply on CEC1', Rachel Furner, 02 Jul 2021
  
  Dear Juan A. Añel,
  Many thanks for your comment and advice, and apologies for this issue. It was picked up by the topical editor, and an amended manuscript uploaded with a suitable zendo link in place of the github link. Somehow though, the original manuscript seems to be the one visible as a preprint, and not this updated version.
  The code is available on zenodo, https://zenodo.org/record/4672260, DOI 10.5281/zenodo.4672260, and I have updated the licence info as suggested. I will make sure this is the link given in any future versions of the manuscript.
  Apologies again for this,
  Rachel
  
  Citation: https://doi.org/10.5194/gmd-2021-132-AC2
CEC1:
'Comment on gmd-2021-132', Juan Antonio Añel, 30 Jun 2021

Dear authors,
After checking your manuscript, it has come to our attention that it does not comply with our Code and Data Policy.

https://www.geoscientific-model-development.net/policies/code_and_data_policy.html

You have archived your code in GitHub. However, GitHub is not a suitable repository. GitHub itself instructs authors to use other alternatives for long-term archival and publishing, such as Zenodo. Therefore, please, publish your code in one of the appropriate repositories.

In this way, before the Discussions stage is closed, you must reply to this comment with the link to the repository for the code and the corresponding DOI.

Also, you must include in a potential reviewed version of your manuscript the modified 'Code and Data Availability' section and the DOI of the code. Also, in the GitHub repository, there is no license listed. If you do not include a license, the code continues to be your property and can not be used by others. Therefore, when uploading the model's code to Zenodo, you could want to choose a free software/open-source (FLOSS) license. We recommend the GPLv3. You only need to include the file 'https://www.gnu.org/licenses/gpl-3.0.txt' as LICENSE.txt with your code. Also, you can choose other options that Zenodo provides: GPLv2, Apache License, MIT License, etc.
Juan A. Añel

Geosc. Mod. Dev. Exec. Editor

Citation: https://doi.org/10.5194/gmd-2021-132-CEC1
- AC2: 'Reply on CEC1', Rachel Furner, 02 Jul 2021
  
  Dear Juan A. Añel,
  Many thanks for your comment and advice, and apologies for this issue. It was picked up by the topical editor, and an amended manuscript uploaded with a suitable zendo link in place of the github link. Somehow though, the original manuscript seems to be the one visible as a preprint, and not this updated version.
  The code is available on zenodo, https://zenodo.org/record/4672260, DOI 10.5281/zenodo.4672260, and I have updated the licence info as suggested. I will make sure this is the link given in any future versions of the manuscript.
  Apologies again for this,
  Rachel
  
  Citation: https://doi.org/10.5194/gmd-2021-132-AC2
RC1:
'Comment on gmd-2021-132', Anonymous Referee #1, 30 Jun 2021

The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2021-132/gmd-2021-132-RC1-supplement.pdf

Citation: https://doi.org/10.5194/gmd-2021-132-RC1
- AC3: 'Reply on RC1', Rachel Furner, 29 Jul 2021
  
  The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2021-132/gmd-2021-132-AC3-supplement.pdf
  
  Citation: https://doi.org/10.5194/gmd-2021-132-AC3
RC2:
'Comment on gmd-2021-132', Anonymous Referee #2, 02 Jul 2021

Recently, the data-driven models have become a hot topic for the atmospheric and oceanic predictions for time scale from synoptic to interannual. Many cases indicate that such models have high predict skills and much less computational resource-consuming. However, as mentioned by the authors, some fundamental questions have no been answered yet, such as if the method can capture the predictability natural of the system and if we can physically explain the results. In the present study, a data-driven model has been developed for predicting the sea surface temperature in the short term. The method they used is a regression model with a non-linear term. The model has been trained using an idealized ocean model dataset as an observing system simulation experiment. They found that the model can predict the SST leading one day, and the dominant variables are also identified. After that, the sensitivity withholding experiments are conducted to identify critical physical variables and processes, like the vertical structure and the non-linear term. In general, this is an interesting and valuable paper and provides helpful information for this kind of data-driven model. Therefore, it is worth to published in GMD after the MINOR revision.

Major questions:

1 I think the most critical problem is the resolution of the model is too coarse (2 degrees) compared with the predicting time scale (1 day). Because the movement of the ocean is much slower than that of the atmosphere, the surrounding points cannot affect the central point for one day. The only fast process is the convection due to instability. That is why the coefficient magnitude of the center point is much larger than other points in Figure 4. If the horizontal resolution is increasing, I guess that the results may also be changed because some processes may transport the signal of around points to the central point during one day. Therefore, I suggest the authors test the sensitivity of the resolution further.

2 In the withholding experiments, we can find that the errors are smaller than the control experiments in some places, such as withholding the information about the vertical structure in Figure 6 and withholding currents in Figure 7. How to understand these results? Is there some information that will bring negative effects or significant errors into the model? Can we find an optimal combination of all this information?

3 The configurations of the model or experiment are not present very clear. Please give more information about the experiment in the manuscript, such as the vertical levels, the time scale of the restoring, the observation of the restoring sea surface temperature and salinity, and the coefficients of the GM scheme.

4 The method of selecting data is also not present very clear. For instance, I do not really understand how to choose the data every 200-day and deal with the 3D variables at the surface. Please say something more about the details.

5 In the present study, the authors only show the results of one-day prediction. I am curious how the model performs when the predicted time scale becomes longer, like 5-day or 10-day. I suggest the authors further discuss the skill of the model for more extended timescale prediction.

Citation: https://doi.org/10.5194/gmd-2021-132-RC2
- AC4: 'Reply on RC2', Rachel Furner, 29 Jul 2021
  
  The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2021-132/gmd-2021-132-AC4-supplement.pdf
  
  Citation: https://doi.org/10.5194/gmd-2021-132-AC4

Status: closed

CC1:
'Comment on gmd-2021-132', Paul PUKITE, 06 Jun 2021
"Specifically, we analyse the coefficients of our regression model and find that the predictions for a grid cell are based

heavily on the density at the surrounding points, and the interaction between temperature at the grid cell and its neighbouring

points. The importance of temperature interaction at surrounding points is representative of advective and diffusive processes

which take place across the domain. The importance of density is in line with the the simulator representing, to some extent,

density driven currents which are responsible for much of the changes in temperature in this GCM configuration"

Density drives climate variability in terms of El Nino/La Nina cycles via the bouyancy of the themocline, where slight density changes above and below the thermocline will impact the effective gravity at the interface. How much of the regression model is sensitive to this narrow equatorial region? Or is it the larger AMO overturning circulation that is a stronger regresssor?
Citation: https://doi.org/10.5194/gmd-2021-132-CC1
- AC1: 'Reply on CC1', Rachel Furner, 21 Jun 2021
  
  Thanks for your comment and the interest in our paper.
  In this instance we’ve intentionally chosen a very idealised and simplified set up, allowing us to more easily analyse the behaviour of the regressor in relation to the simulator. This means the underlying MITgcm simulation which the regressor is learning has constant surface forcing, so the training dataset has no El Nino/La Nina cycle, and hence the regressor (which is intended to mimic this dataset) also won’t show any signal related to El Nino/La Nina.
  It is however an interesting question as to how much the regressor would be able to learn from a more realistic set up which did include El Nino/La Nina, given the relatively small geographical extent of this. In our set up the regressor is learning a single equation, to be applied at each grid box in the domain (rather than learning specifics about the dynamics of different regions) and it is learning the (relatively) small scale, local dynamics, rather than large scale patterns such as El Nino/La Nina. In this sense it is very similar to the way that traditional GCMs make their predictions – values for a variable at a grid box are changed according to a consistent algorithm which takes as inputs the values at the grid cell and its surrounding cells. This grid-cell scale physics then leads to the wider circulation features such as El Nino/La Nina rather than these being incorporated into the GCM directly.
  Given this we might reasonably expect the regressor to capture the cell-level dynamics which lead to larger features such as El Nino/La Nina in the same way as GCMs do. This is however dependent on how closely the regressor learns the physics of the system.
  
  Citation: https://doi.org/10.5194/gmd-2021-132-AC1
- AC2: 'Reply on CEC1', Rachel Furner, 02 Jul 2021
  
  Dear Juan A. Añel,
  Many thanks for your comment and advice, and apologies for this issue. It was picked up by the topical editor, and an amended manuscript uploaded with a suitable zendo link in place of the github link. Somehow though, the original manuscript seems to be the one visible as a preprint, and not this updated version.
  The code is available on zenodo, https://zenodo.org/record/4672260, DOI 10.5281/zenodo.4672260, and I have updated the licence info as suggested. I will make sure this is the link given in any future versions of the manuscript.
  Apologies again for this,
  Rachel
  
  Citation: https://doi.org/10.5194/gmd-2021-132-AC2
CEC1:
'Comment on gmd-2021-132', Juan Antonio Añel, 30 Jun 2021

Dear authors,
After checking your manuscript, it has come to our attention that it does not comply with our Code and Data Policy.

https://www.geoscientific-model-development.net/policies/code_and_data_policy.html

You have archived your code in GitHub. However, GitHub is not a suitable repository. GitHub itself instructs authors to use other alternatives for long-term archival and publishing, such as Zenodo. Therefore, please, publish your code in one of the appropriate repositories.

In this way, before the Discussions stage is closed, you must reply to this comment with the link to the repository for the code and the corresponding DOI.

Also, you must include in a potential reviewed version of your manuscript the modified 'Code and Data Availability' section and the DOI of the code. Also, in the GitHub repository, there is no license listed. If you do not include a license, the code continues to be your property and can not be used by others. Therefore, when uploading the model's code to Zenodo, you could want to choose a free software/open-source (FLOSS) license. We recommend the GPLv3. You only need to include the file 'https://www.gnu.org/licenses/gpl-3.0.txt' as LICENSE.txt with your code. Also, you can choose other options that Zenodo provides: GPLv2, Apache License, MIT License, etc.
Juan A. Añel

Geosc. Mod. Dev. Exec. Editor

Citation: https://doi.org/10.5194/gmd-2021-132-CEC1
- AC2: 'Reply on CEC1', Rachel Furner, 02 Jul 2021
  
  Dear Juan A. Añel,
  Many thanks for your comment and advice, and apologies for this issue. It was picked up by the topical editor, and an amended manuscript uploaded with a suitable zendo link in place of the github link. Somehow though, the original manuscript seems to be the one visible as a preprint, and not this updated version.
  The code is available on zenodo, https://zenodo.org/record/4672260, DOI 10.5281/zenodo.4672260, and I have updated the licence info as suggested. I will make sure this is the link given in any future versions of the manuscript.
  Apologies again for this,
  Rachel
  
  Citation: https://doi.org/10.5194/gmd-2021-132-AC2
RC1:
'Comment on gmd-2021-132', Anonymous Referee #1, 30 Jun 2021

The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2021-132/gmd-2021-132-RC1-supplement.pdf

Citation: https://doi.org/10.5194/gmd-2021-132-RC1
- AC3: 'Reply on RC1', Rachel Furner, 29 Jul 2021
  
  The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2021-132/gmd-2021-132-AC3-supplement.pdf
  
  Citation: https://doi.org/10.5194/gmd-2021-132-AC3
RC2:
'Comment on gmd-2021-132', Anonymous Referee #2, 02 Jul 2021

Recently, the data-driven models have become a hot topic for the atmospheric and oceanic predictions for time scale from synoptic to interannual. Many cases indicate that such models have high predict skills and much less computational resource-consuming. However, as mentioned by the authors, some fundamental questions have no been answered yet, such as if the method can capture the predictability natural of the system and if we can physically explain the results. In the present study, a data-driven model has been developed for predicting the sea surface temperature in the short term. The method they used is a regression model with a non-linear term. The model has been trained using an idealized ocean model dataset as an observing system simulation experiment. They found that the model can predict the SST leading one day, and the dominant variables are also identified. After that, the sensitivity withholding experiments are conducted to identify critical physical variables and processes, like the vertical structure and the non-linear term. In general, this is an interesting and valuable paper and provides helpful information for this kind of data-driven model. Therefore, it is worth to published in GMD after the MINOR revision.

Major questions:

1 I think the most critical problem is the resolution of the model is too coarse (2 degrees) compared with the predicting time scale (1 day). Because the movement of the ocean is much slower than that of the atmosphere, the surrounding points cannot affect the central point for one day. The only fast process is the convection due to instability. That is why the coefficient magnitude of the center point is much larger than other points in Figure 4. If the horizontal resolution is increasing, I guess that the results may also be changed because some processes may transport the signal of around points to the central point during one day. Therefore, I suggest the authors test the sensitivity of the resolution further.

2 In the withholding experiments, we can find that the errors are smaller than the control experiments in some places, such as withholding the information about the vertical structure in Figure 6 and withholding currents in Figure 7. How to understand these results? Is there some information that will bring negative effects or significant errors into the model? Can we find an optimal combination of all this information?

3 The configurations of the model or experiment are not present very clear. Please give more information about the experiment in the manuscript, such as the vertical levels, the time scale of the restoring, the observation of the restoring sea surface temperature and salinity, and the coefficients of the GM scheme.

4 The method of selecting data is also not present very clear. For instance, I do not really understand how to choose the data every 200-day and deal with the 3D variables at the surface. Please say something more about the details.

5 In the present study, the authors only show the results of one-day prediction. I am curious how the model performs when the predicted time scale becomes longer, like 5-day or 10-day. I suggest the authors further discuss the skill of the model for more extended timescale prediction.

Citation: https://doi.org/10.5194/gmd-2021-132-RC2
- AC4: 'Reply on RC2', Rachel Furner, 29 Jul 2021
  
  The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2021-132/gmd-2021-132-AC4-supplement.pdf
  
  Citation: https://doi.org/10.5194/gmd-2021-132-AC4

Rachel Furner, Peter Haynes, Dave Munday, Brooks Paige, Daniel C. Jones, and Emily Shuckburgh

Viewed

Total article views: 2,056 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	BibTeX	EndNote
1,435	553	68	2,056	36	36

HTML: 1,435
PDF: 553
XML: 68
Total: 2,056
BibTeX: 36
EndNote: 36

Views and downloads (calculated since 02 Jun 2021)

Month	HTML	PDF	XML	Total
Jun 2021	535	71	5	611
Jul 2021	136	51	10	197
Aug 2021	49	12	0	61
Sep 2021	36	31	0	67
Oct 2021	32	66	0	98
Nov 2021	36	44	0	80
Dec 2021	42	20	7	69
Jan 2022	27	8	1	36
Feb 2022	24	6	1	31
Mar 2022	25	4	2	31
Apr 2022	21	4	1	26
May 2022	8	9	3	20
Jun 2022	9	5	1	15
Jul 2022	7	2	1	10
Aug 2022	8	8	1	17
Sep 2022	23	5	0	28
Oct 2022	14	3	1	18
Nov 2022	14	9	0	23
Dec 2022	10	9	0	19
Jan 2023	20	18	1	39
Feb 2023	8	1	0	9
Mar 2023	15	5	1	21
Apr 2023	12	6	0	18
May 2023	12	7	1	20
Jun 2023	11	19	5	35
Jul 2023	10	15	1	26
Aug 2023	10	15	0	25
Sep 2023	29	6	1	36
Oct 2023	35	8	0	43
Nov 2023	15	3	0	18
Dec 2023	21	4	1	26
Jan 2024	11	6	1	18
Feb 2024	19	12	3	34
Mar 2024	27	18	5	50
Apr 2024	25	5	3	33
May 2024	11	1	3	15
Jun 2024	13	8	2	23
Jul 2024	9	7	3	19
Aug 2024	16	4	2	22
Sep 2024	11	5	0	16
Oct 2024	10	8	0	18
Nov 2024	24	3	1	28
Dec 2024	5	2	0	7

Cumulative views and downloads (calculated since 02 Jun 2021)

Month	HTML	PDF	XML	Total
Jun 2021	535	71	5	611
Jul 2021	136	51	10	197
Aug 2021	49	12	0	61
Sep 2021	36	31	0	67
Oct 2021	32	66	0	98
Nov 2021	36	44	0	80
Dec 2021	42	20	7	69
Jan 2022	27	8	1	36
Feb 2022	24	6	1	31
Mar 2022	25	4	2	31
Apr 2022	21	4	1	26
May 2022	8	9	3	20
Jun 2022	9	5	1	15
Jul 2022	7	2	1	10
Aug 2022	8	8	1	17
Sep 2022	23	5	0	28
Oct 2022	14	3	1	18
Nov 2022	14	9	0	23
Dec 2022	10	9	0	19
Jan 2023	20	18	1	39
Feb 2023	8	1	0	9
Mar 2023	15	5	1	21
Apr 2023	12	6	0	18
May 2023	12	7	1	20
Jun 2023	11	19	5	35
Jul 2023	10	15	1	26
Aug 2023	10	15	0	25
Sep 2023	29	6	1	36
Oct 2023	35	8	0	43
Nov 2023	15	3	0	18
Dec 2023	21	4	1	26
Jan 2024	11	6	1	18
Feb 2024	19	12	3	34
Mar 2024	27	18	5	50
Apr 2024	25	5	3	33
May 2024	11	1	3	15
Jun 2024	13	8	2	23
Jul 2024	9	7	3	19
Aug 2024	16	4	2	22
Sep 2024	11	5	0	16
Oct 2024	10	8	0	18
Nov 2024	24	3	1	28
Dec 2024	5	2	0	7

Viewed (geographical distribution)

Total article views: 1,863 (including HTML, PDF, and XML) Thereof 1,863 with geography defined and 0 with unknown origin.

Country	#	Views	%

Latest update: 13 Dec 2024

Short summary

Traditional weather & climate models are built from physics-based equations, while data-driven models are built from patterns found in datasets using Machine Learning or statistics. There is growing interest in using data-driven models for weather & climate prediction, but confidence in their use depends on understanding the patterns they're finding. We look at this with a simple regression model of ocean temperature and see the patterns found by the regression model are similar to the physics.


Total:	0
HTML:	0
PDF:	0
XML:	0