the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Sensitivity analysis of a data-driven model of ocean temperature
Abstract. There has been much recent interest in developing data-driven models for weather and climate predictions. These have shown reasonable success in modelling atmospheric dynamics over short time scales, however there are open questions regarding the sensitivity and robustness of these models. Using model interpretation techniques to better understand how data-driven models are making predictions is critical to developing trust in these alternative prediction systems. We develop a simple regression model of ocean temperature evolution, Ocean Temperature Regressor v1.0, and investigate its sensitivity to improve understanding of whether data-driven models are capable of learning the complex underlying dynamics of the systems being modelled.
We investigate model sensitivity in a variety of ways and find that Ocean Temperature Regressor v1.0 behaves in ways which are, for the most part, in line with our knowledge of the ocean system being modelled. Specifically we see that the regressor heavily bases its forecasts on, and is dependent on, variables which we know are key to the physical dynamics inherent in the system, such as the currents and density. By contrast, inputs to the regressor which have limited direct dynamic impact, such as location, are not heavily used by the regressor. We also find that the regression model requires non-linear interactions between inputs in order to show any meaningful predictive skill – in line with out knowledge of the highly nonlinear dynamics of the ocean. Further sensitivity analysis is carried out to interpret that the ways in which certain variables are used by the regression model. Results here are again mostly in line with our physical knowledge of the system, for example, we see that information about the vertical profile of the water column reduces errors in areas associated with convective activity, and information about the currents is used by the regressor to reduce errors in regions dominated by advective processes.
Our results show that even a simple regression model is capable of “learning” much of the physical dynamics inherent in the ocean system being modelled, which gives promise for the sensitivity and generalisability of data-driven models more generally.
- Preprint
(2980 KB) - Metadata XML
- BibTeX
- EndNote
Status: closed
-
CC1: 'Comment on gmd-2021-132', Paul PUKITE, 06 Jun 2021
- "Specifically, we analyse the coefficients of our regression model and find that the predictions for a grid cell are based
heavily on the density at the surrounding points, and the interaction between temperature at the grid cell and its neighbouring
points. The importance of temperature interaction at surrounding points is representative of advective and diffusive processes
which take place across the domain. The importance of density is in line with the the simulator representing, to some extent,
density driven currents which are responsible for much of the changes in temperature in this GCM configuration"
Density drives climate variability in terms of El Nino/La Nina cycles via the bouyancy of the themocline, where slight density changes above and below the thermocline will impact the effective gravity at the interface. How much of the regression model is sensitive to this narrow equatorial region? Or is it the larger AMO overturning circulation that is a stronger regresssor?
Citation: https://doi.org/10.5194/gmd-2021-132-CC1 -
AC1: 'Reply on CC1', Rachel Furner, 21 Jun 2021
Thanks for your comment and the interest in our paper.
In this instance we’ve intentionally chosen a very idealised and simplified set up, allowing us to more easily analyse the behaviour of the regressor in relation to the simulator. This means the underlying MITgcm simulation which the regressor is learning has constant surface forcing, so the training dataset has no El Nino/La Nina cycle, and hence the regressor (which is intended to mimic this dataset) also won’t show any signal related to El Nino/La Nina.
It is however an interesting question as to how much the regressor would be able to learn from a more realistic set up which did include El Nino/La Nina, given the relatively small geographical extent of this. In our set up the regressor is learning a single equation, to be applied at each grid box in the domain (rather than learning specifics about the dynamics of different regions) and it is learning the (relatively) small scale, local dynamics, rather than large scale patterns such as El Nino/La Nina. In this sense it is very similar to the way that traditional GCMs make their predictions – values for a variable at a grid box are changed according to a consistent algorithm which takes as inputs the values at the grid cell and its surrounding cells. This grid-cell scale physics then leads to the wider circulation features such as El Nino/La Nina rather than these being incorporated into the GCM directly.
Given this we might reasonably expect the regressor to capture the cell-level dynamics which lead to larger features such as El Nino/La Nina in the same way as GCMs do. This is however dependent on how closely the regressor learns the physics of the system.
Citation: https://doi.org/10.5194/gmd-2021-132-AC1 -
AC2: 'Reply on CEC1', Rachel Furner, 02 Jul 2021
Dear Juan A. Añel,
Many thanks for your comment and advice, and apologies for this issue. It was picked up by the topical editor, and an amended manuscript uploaded with a suitable zendo link in place of the github link. Somehow though, the original manuscript seems to be the one visible as a preprint, and not this updated version.
The code is available on zenodo, https://zenodo.org/record/4672260, DOI 10.5281/zenodo.4672260, and I have updated the licence info as suggested. I will make sure this is the link given in any future versions of the manuscript.
Apologies again for this,
Rachel
Citation: https://doi.org/10.5194/gmd-2021-132-AC2
- "Specifically, we analyse the coefficients of our regression model and find that the predictions for a grid cell are based
-
CEC1: 'Comment on gmd-2021-132', Juan Antonio Añel, 30 Jun 2021
Dear authors,
After checking your manuscript, it has come to our attention that it does not comply with our Code and Data Policy.
https://www.geoscientific-model-development.net/policies/code_and_data_policy.html
You have archived your code in GitHub. However, GitHub is not a suitable repository. GitHub itself instructs authors to use other alternatives for long-term archival and publishing, such as Zenodo. Therefore, please, publish your code in one of the appropriate repositories.
In this way, before the Discussions stage is closed, you must reply to this comment with the link to the repository for the code and the corresponding DOI.
Also, you must include in a potential reviewed version of your manuscript the modified 'Code and Data Availability' section and the DOI of the code. Also, in the GitHub repository, there is no license listed. If you do not include a license, the code continues to be your property and can not be used by others. Therefore, when uploading the model's code to Zenodo, you could want to choose a free software/open-source (FLOSS) license. We recommend the GPLv3. You only need to include the file 'https://www.gnu.org/licenses/gpl-3.0.txt' as LICENSE.txt with your code. Also, you can choose other options that Zenodo provides: GPLv2, Apache License, MIT License, etc.Juan A. Añel
Geosc. Mod. Dev. Exec. EditorCitation: https://doi.org/10.5194/gmd-2021-132-CEC1 -
AC2: 'Reply on CEC1', Rachel Furner, 02 Jul 2021
Dear Juan A. Añel,
Many thanks for your comment and advice, and apologies for this issue. It was picked up by the topical editor, and an amended manuscript uploaded with a suitable zendo link in place of the github link. Somehow though, the original manuscript seems to be the one visible as a preprint, and not this updated version.
The code is available on zenodo, https://zenodo.org/record/4672260, DOI 10.5281/zenodo.4672260, and I have updated the licence info as suggested. I will make sure this is the link given in any future versions of the manuscript.
Apologies again for this,
Rachel
Citation: https://doi.org/10.5194/gmd-2021-132-AC2
-
AC2: 'Reply on CEC1', Rachel Furner, 02 Jul 2021
-
RC1: 'Comment on gmd-2021-132', Anonymous Referee #1, 30 Jun 2021
The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2021-132/gmd-2021-132-RC1-supplement.pdf
-
AC3: 'Reply on RC1', Rachel Furner, 29 Jul 2021
The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2021-132/gmd-2021-132-AC3-supplement.pdf
-
AC3: 'Reply on RC1', Rachel Furner, 29 Jul 2021
-
RC2: 'Comment on gmd-2021-132', Anonymous Referee #2, 02 Jul 2021
Recently, the data-driven models have become a hot topic for the atmospheric and oceanic predictions for time scale from synoptic to interannual. Many cases indicate that such models have high predict skills and much less computational resource-consuming. However, as mentioned by the authors, some fundamental questions have no been answered yet, such as if the method can capture the predictability natural of the system and if we can physically explain the results. In the present study, a data-driven model has been developed for predicting the sea surface temperature in the short term. The method they used is a regression model with a non-linear term. The model has been trained using an idealized ocean model dataset as an observing system simulation experiment. They found that the model can predict the SST leading one day, and the dominant variables are also identified. After that, the sensitivity withholding experiments are conducted to identify critical physical variables and processes, like the vertical structure and the non-linear term. In general, this is an interesting and valuable paper and provides helpful information for this kind of data-driven model. Therefore, it is worth to published in GMD after the MINOR revision.
Major questions:
1 I think the most critical problem is the resolution of the model is too coarse (2 degrees) compared with the predicting time scale (1 day). Because the movement of the ocean is much slower than that of the atmosphere, the surrounding points cannot affect the central point for one day. The only fast process is the convection due to instability. That is why the coefficient magnitude of the center point is much larger than other points in Figure 4. If the horizontal resolution is increasing, I guess that the results may also be changed because some processes may transport the signal of around points to the central point during one day. Therefore, I suggest the authors test the sensitivity of the resolution further.2 In the withholding experiments, we can find that the errors are smaller than the control experiments in some places, such as withholding the information about the vertical structure in Figure 6 and withholding currents in Figure 7. How to understand these results? Is there some information that will bring negative effects or significant errors into the model? Can we find an optimal combination of all this information?
3 The configurations of the model or experiment are not present very clear. Please give more information about the experiment in the manuscript, such as the vertical levels, the time scale of the restoring, the observation of the restoring sea surface temperature and salinity, and the coefficients of the GM scheme.
4 The method of selecting data is also not present very clear. For instance, I do not really understand how to choose the data every 200-day and deal with the 3D variables at the surface. Please say something more about the details.
5 In the present study, the authors only show the results of one-day prediction. I am curious how the model performs when the predicted time scale becomes longer, like 5-day or 10-day. I suggest the authors further discuss the skill of the model for more extended timescale prediction.
Citation: https://doi.org/10.5194/gmd-2021-132-RC2 -
AC4: 'Reply on RC2', Rachel Furner, 29 Jul 2021
The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2021-132/gmd-2021-132-AC4-supplement.pdf
-
AC4: 'Reply on RC2', Rachel Furner, 29 Jul 2021
Status: closed
-
CC1: 'Comment on gmd-2021-132', Paul PUKITE, 06 Jun 2021
- "Specifically, we analyse the coefficients of our regression model and find that the predictions for a grid cell are based
heavily on the density at the surrounding points, and the interaction between temperature at the grid cell and its neighbouring
points. The importance of temperature interaction at surrounding points is representative of advective and diffusive processes
which take place across the domain. The importance of density is in line with the the simulator representing, to some extent,
density driven currents which are responsible for much of the changes in temperature in this GCM configuration"
Density drives climate variability in terms of El Nino/La Nina cycles via the bouyancy of the themocline, where slight density changes above and below the thermocline will impact the effective gravity at the interface. How much of the regression model is sensitive to this narrow equatorial region? Or is it the larger AMO overturning circulation that is a stronger regresssor?
Citation: https://doi.org/10.5194/gmd-2021-132-CC1 -
AC1: 'Reply on CC1', Rachel Furner, 21 Jun 2021
Thanks for your comment and the interest in our paper.
In this instance we’ve intentionally chosen a very idealised and simplified set up, allowing us to more easily analyse the behaviour of the regressor in relation to the simulator. This means the underlying MITgcm simulation which the regressor is learning has constant surface forcing, so the training dataset has no El Nino/La Nina cycle, and hence the regressor (which is intended to mimic this dataset) also won’t show any signal related to El Nino/La Nina.
It is however an interesting question as to how much the regressor would be able to learn from a more realistic set up which did include El Nino/La Nina, given the relatively small geographical extent of this. In our set up the regressor is learning a single equation, to be applied at each grid box in the domain (rather than learning specifics about the dynamics of different regions) and it is learning the (relatively) small scale, local dynamics, rather than large scale patterns such as El Nino/La Nina. In this sense it is very similar to the way that traditional GCMs make their predictions – values for a variable at a grid box are changed according to a consistent algorithm which takes as inputs the values at the grid cell and its surrounding cells. This grid-cell scale physics then leads to the wider circulation features such as El Nino/La Nina rather than these being incorporated into the GCM directly.
Given this we might reasonably expect the regressor to capture the cell-level dynamics which lead to larger features such as El Nino/La Nina in the same way as GCMs do. This is however dependent on how closely the regressor learns the physics of the system.
Citation: https://doi.org/10.5194/gmd-2021-132-AC1 -
AC2: 'Reply on CEC1', Rachel Furner, 02 Jul 2021
Dear Juan A. Añel,
Many thanks for your comment and advice, and apologies for this issue. It was picked up by the topical editor, and an amended manuscript uploaded with a suitable zendo link in place of the github link. Somehow though, the original manuscript seems to be the one visible as a preprint, and not this updated version.
The code is available on zenodo, https://zenodo.org/record/4672260, DOI 10.5281/zenodo.4672260, and I have updated the licence info as suggested. I will make sure this is the link given in any future versions of the manuscript.
Apologies again for this,
Rachel
Citation: https://doi.org/10.5194/gmd-2021-132-AC2
- "Specifically, we analyse the coefficients of our regression model and find that the predictions for a grid cell are based
-
CEC1: 'Comment on gmd-2021-132', Juan Antonio Añel, 30 Jun 2021
Dear authors,
After checking your manuscript, it has come to our attention that it does not comply with our Code and Data Policy.
https://www.geoscientific-model-development.net/policies/code_and_data_policy.html
You have archived your code in GitHub. However, GitHub is not a suitable repository. GitHub itself instructs authors to use other alternatives for long-term archival and publishing, such as Zenodo. Therefore, please, publish your code in one of the appropriate repositories.
In this way, before the Discussions stage is closed, you must reply to this comment with the link to the repository for the code and the corresponding DOI.
Also, you must include in a potential reviewed version of your manuscript the modified 'Code and Data Availability' section and the DOI of the code. Also, in the GitHub repository, there is no license listed. If you do not include a license, the code continues to be your property and can not be used by others. Therefore, when uploading the model's code to Zenodo, you could want to choose a free software/open-source (FLOSS) license. We recommend the GPLv3. You only need to include the file 'https://www.gnu.org/licenses/gpl-3.0.txt' as LICENSE.txt with your code. Also, you can choose other options that Zenodo provides: GPLv2, Apache License, MIT License, etc.Juan A. Añel
Geosc. Mod. Dev. Exec. EditorCitation: https://doi.org/10.5194/gmd-2021-132-CEC1 -
AC2: 'Reply on CEC1', Rachel Furner, 02 Jul 2021
Dear Juan A. Añel,
Many thanks for your comment and advice, and apologies for this issue. It was picked up by the topical editor, and an amended manuscript uploaded with a suitable zendo link in place of the github link. Somehow though, the original manuscript seems to be the one visible as a preprint, and not this updated version.
The code is available on zenodo, https://zenodo.org/record/4672260, DOI 10.5281/zenodo.4672260, and I have updated the licence info as suggested. I will make sure this is the link given in any future versions of the manuscript.
Apologies again for this,
Rachel
Citation: https://doi.org/10.5194/gmd-2021-132-AC2
-
AC2: 'Reply on CEC1', Rachel Furner, 02 Jul 2021
-
RC1: 'Comment on gmd-2021-132', Anonymous Referee #1, 30 Jun 2021
The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2021-132/gmd-2021-132-RC1-supplement.pdf
-
AC3: 'Reply on RC1', Rachel Furner, 29 Jul 2021
The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2021-132/gmd-2021-132-AC3-supplement.pdf
-
AC3: 'Reply on RC1', Rachel Furner, 29 Jul 2021
-
RC2: 'Comment on gmd-2021-132', Anonymous Referee #2, 02 Jul 2021
Recently, the data-driven models have become a hot topic for the atmospheric and oceanic predictions for time scale from synoptic to interannual. Many cases indicate that such models have high predict skills and much less computational resource-consuming. However, as mentioned by the authors, some fundamental questions have no been answered yet, such as if the method can capture the predictability natural of the system and if we can physically explain the results. In the present study, a data-driven model has been developed for predicting the sea surface temperature in the short term. The method they used is a regression model with a non-linear term. The model has been trained using an idealized ocean model dataset as an observing system simulation experiment. They found that the model can predict the SST leading one day, and the dominant variables are also identified. After that, the sensitivity withholding experiments are conducted to identify critical physical variables and processes, like the vertical structure and the non-linear term. In general, this is an interesting and valuable paper and provides helpful information for this kind of data-driven model. Therefore, it is worth to published in GMD after the MINOR revision.
Major questions:
1 I think the most critical problem is the resolution of the model is too coarse (2 degrees) compared with the predicting time scale (1 day). Because the movement of the ocean is much slower than that of the atmosphere, the surrounding points cannot affect the central point for one day. The only fast process is the convection due to instability. That is why the coefficient magnitude of the center point is much larger than other points in Figure 4. If the horizontal resolution is increasing, I guess that the results may also be changed because some processes may transport the signal of around points to the central point during one day. Therefore, I suggest the authors test the sensitivity of the resolution further.2 In the withholding experiments, we can find that the errors are smaller than the control experiments in some places, such as withholding the information about the vertical structure in Figure 6 and withholding currents in Figure 7. How to understand these results? Is there some information that will bring negative effects or significant errors into the model? Can we find an optimal combination of all this information?
3 The configurations of the model or experiment are not present very clear. Please give more information about the experiment in the manuscript, such as the vertical levels, the time scale of the restoring, the observation of the restoring sea surface temperature and salinity, and the coefficients of the GM scheme.
4 The method of selecting data is also not present very clear. For instance, I do not really understand how to choose the data every 200-day and deal with the 3D variables at the surface. Please say something more about the details.
5 In the present study, the authors only show the results of one-day prediction. I am curious how the model performs when the predicted time scale becomes longer, like 5-day or 10-day. I suggest the authors further discuss the skill of the model for more extended timescale prediction.
Citation: https://doi.org/10.5194/gmd-2021-132-RC2 -
AC4: 'Reply on RC2', Rachel Furner, 29 Jul 2021
The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2021-132/gmd-2021-132-AC4-supplement.pdf
-
AC4: 'Reply on RC2', Rachel Furner, 29 Jul 2021
Viewed
HTML | XML | Total | BibTeX | EndNote | |
---|---|---|---|---|---|
1,433 | 551 | 68 | 2,052 | 36 | 36 |
- HTML: 1,433
- PDF: 551
- XML: 68
- Total: 2,052
- BibTeX: 36
- EndNote: 36
Viewed (geographical distribution)
Country | # | Views | % |
---|
Total: | 0 |
HTML: | 0 |
PDF: | 0 |
XML: | 0 |
- 1