Stable climate simulations using a realistic general circulation model with neural network parameterizations for atmospheric moist physics and radiation processes

Wang, Xin; Han, Yilun; Xue, Wei; Yang, Guangwen; Zhang, Guang J.

doi:https://doi.org/10.5194/gmd-15-3923-2022

Articles | Volume 15, issue 9

https://doi.org/10.5194/gmd-15-3923-2022

© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/gmd-15-3923-2022

© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 15, issue 9

Development and technical paper

|

16 May 2022

Development and technical paper |

| 16 May 2022

Stable climate simulations using a realistic general circulation model with neural network parameterizations for atmospheric moist physics and radiation processes

Xin Wang, Yilun Han, Wei Xue, Guangwen Yang, and Guang J. Zhang

Download

Final revised paper (published on 16 May 2022)
Supplement to the final revised paper
Preprint (discussion started on 29 Sep 2021)

Interactive discussion

Status: closed

CEC1:
'Comment on gmd-2021-299', Juan Antonio Añel, 25 Oct 2021

Dear authors,
After checking your manuscript, it has come to our attention that it does not comply with our Code and Data Policy.

https://www.geoscientific-model-development.net/policies/code_and_data_policy.html
First of all, the SPCAM v2 model is only available from a SVN repository and the access is restricted. We can not accept this. You must publish the code of the model in one of the appropriate repositories (e.g., Zenodo) without restrictions.

Secondly, the coupler to use the DNN with the model seems to be one of the most relevant parts of your work; however, in the NNCAM file in the Zenodo repository, it is not clear if it is included. I guess that it could be in the nncam-couple directory, but the fact is that in the main directory, there is no description about what this file contains, what each script does, etc. Please, add a file with the information. The structure is similar to the one of the CCSM or CESM, but it is necessary to explain better for readers not familiar with these models.

Also, when using machine learning techniques that include training, the data used to train the neural network are of the utmost importance for the reproducibility of the experiments. Therefore, please, share the data that you have used in the Zenodo repository.

Please, address all these issues as soon as possible, as this information must be available for the Discussions stage. Also, remember that you must include the modified 'Code and Data Availability' with all the new information in a potential reviewed version of your manuscript.
Best regards,
Juan A. Añel
Geosc. Mod. Dev. Executive Editor

Citation: https://doi.org/10.5194/gmd-2021-299-CEC1
- AC1: 'Reply on CEC1', Xin Wang, 25 Oct 2021
  
  Dear Dr. Añel,
  Many thanks for the comment.
  First, we are sorry for the inconvenience when accessing the SPCAM v2 model. Here we provide SPCAM v2 model directly in Zenodo next to the NNCAM. And we have submitted a proper description of the case used to generate target data for training, validation and testing.
  Secondly, we thanks for your attention on our work. Yes, the DNN-GCM Coupler is one of the most important parts in our work. We are sorry that we did not describe it in detail. Now we provide a sophisticated README file in our Zenodo repository.
  We are fully aware the responsibility of scientists to share the original data, and we are preparing to do so. Right now, duty to the storage quota of Zenodo, we can only provide one year of the SPCAM simulation with all considered variables in training. That is the maximum data we can provide, even after we apply for a 100GB storage quota. We believe that these data is enough to reduplicate the neural network training. If the readers want more data, we recommend they install the provided SPCAM and run the suggest case.
  It will take a few days to fully upload all the data and codes in Zenodo. Right now we provide you source codes of SPCAM and NNCAM include proper descriptions file.
  They are attached in the reply.
  Best regards,
  Xin Wang
  
  Citation: https://doi.org/10.5194/gmd-2021-299-AC1
- AC2: 'Reply on CEC1', Xin Wang, 01 Nov 2021
  
  Dear Dr. Añel,
  Sorry for the delay. Right now, we are glad to say all required data and files are uploaded into zenodo depositories: https://doi.org/10.5281/zenodo.5596273 and https://doi.org/10.5281/zenodo.5625616. The first is for the source codes of SPCAMv2 and NNCAM, while the second is for training data. Please check them and contact us if you have further questions.
  Both new links will be put in the modified ‘Code and data availability’ in the reviewed version of our manuscript.
  Best Regards.
  
  Xin Wang.
  
  Citation: https://doi.org/10.5194/gmd-2021-299-AC2
RC1:
'Review on gmd-2021-299', Anonymous Referee #1, 04 Nov 2021

See attached file for full review

Citation: https://doi.org/10.5194/gmd-2021-299-RC1
- AC4: 'Reply on RC1', Xin Wang, 10 Jan 2022
  
  The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2021-299/gmd-2021-299-AC4-supplement.pdf
  
  Citation: https://doi.org/10.5194/gmd-2021-299-AC4
RC2:
'Comment on gmd-2021-299', Anonymous Referee #2, 07 Nov 2021
In this manuscript the authors use neural networks to emulate the grid-box mean output of a superparametrization scheme, which predicts the sub-grid tendencies for moist physics and radiative heating. After a period of offline training the authors develop a new coupling approach for online testing. In online testing they find some evidence of improvements over the existing CAM5, i.e. a closer fit to the SPCAM approach.

There are several interesting ideas in this manuscript and some impressive technical developments in the coupling framework. However, I do not currently feel the manuscript is near acceptance for publication. My main issue is that the online testing analysis is not consistent, and does not persuade the reader that NNCAM is an improvement over the existing CAM5 model. Given that NNCAM is slower than the normal CAM5 parametrizations I think it is important to show that NNCAM provides an improvement. If this is not possible, then instead the authors could focus more effort on establishing whether any offline metrics provide a better indicator of online stability. Below I will detail my comments further. I hope the authors will take these on board, as I think this manuscript could make for an interesting and useful paper.

The section on online performance analysis is a weak point. I think it is important to standardise the measurement periods used by the CAM5, SPCAM and NNCAM. Showing CAM5 and NNCAM as deviations away from the "truth" of SPCAM would ease the process of comparison. In many of the figures it is unclear that NNCAM is an improvement on CAM5, which begs the question of the purpose of the networks. I also think that showing plots of global metrics against time would help identify drift in the models. Given that this paper has a climate motivation, examining this behaviour seems crucial. I also think there is insufficient analysis of the effects of emulating radiative heating. It would be interesting to see some global maps of average 2m temperature to see the effects of the surface fluxes and near-surface heating rates.

The authors highlight that online stability is not a given for coupling of parametrization schemes. This is a really interesting and important point for this field of study. However the authors' proposed solution is trial and error, suggesting that short term stability is a good predictor of long term stability. I would like to see more detailed analysis of whether there are good offline measures that can guide online stability. The authors suggest that improving R2 scores are not fully correlated with stability. Can one find a different metric that is better correlated with stability, analysing the results the authors have already conducted? I would be interested to see if mean-squared-error, mean bias or some measure of worst error were better predictors. If I understand the work correctly, you train four networks in your SPCAM. When you test stability are you swapping these four networks individually, or swapping all four together? This might shed light on which components were more important for stability. I think studying these points could provide great insight into the problem.

If I understand the training/validation/testing split correctly, these are random subsets in space and time from the 1998/9 dataset. If so, I do not think this is a safe method for ensuring no overfitting, as this does not take into account spatial/temporal correlations. I think the total dataset should be split by time only, with temporal gaps between training, validation and testing to ensure independence. This might explain why NN with better R2 values provide less stable answers, if there is overfitting on the dataset.

There is very little discussion about the benefits and downsides to superparametrization. It is my understanding that there is very limited (if any) evidence that superparametrization actually improves model climate versus typical parametrization schemes. I think it is worth stating this, or if the authors disagree, provide citations.

Are the only parametrizations within the CAM model those in the superparametrization? e.g. there is no parametrization for sub-grid orographic gravity waves.

I suggest re-ordering manuscript to explain coupling before explaining results. The results section makes reference to coupled testing without explaining how this is achieved.

L135: Where does the variability originate in the CRMs? Are they initialised with different perturbations of the larger-scale conditions? If there is stochasticity in the system? It would be good to state this if true.

L166: "as output the NN-Parameterization". I think this should be "as outputs from the NN-Parameterization".

L167 "is critical to improve the performance of the NNCAM". I could find no further discussion of this. It sounds like a very interesting point. Please expand.

L190: Are you training to maximise R2? If not, what is your function to minimise/maximise?

L195: Have you tested this theory of mutual interference? I would have thought that training two different models to predict the TOA and surface fluxes would introduce physical inconsistencies. These are not separate pieces of physics.

L214: "a well-fit is necessary". This was unclear and could be better written.

L229: Is the "best performance" network based upon the best performance in offline or online testing?

L242: The online coupler sounds like an interesting solution of value to the wider scientific community. Are the authors planning to share this as a stand-alone piece of software?

L278: I do not understand "reaches half the speed of CAM5". Are the authors comparing to the speed of CAM5 with the normal parametrization schemes? By half the speed to they mean it will take twice the time to simulate the same period?

L279: Have the authors profiled how much time is spent communicating data versus doing ML inference? This would be very interesting to see.

L280: If I have understood correctly the authors carry out the online testing on the same time period that the NN was trained on. Has any effort been made to ensure independence between the training and testing data?

L305: "tunned" -> "tuned"

L320: The authors run for 10 years but only analyse 4 years of data. So their only expectation of the final 5 years is for the model to not crash. I do not think this is an appropriately strict assessment for their NN models. I think examining model drift is exactly the important test of a NN. If not, what is the purpose of the model that the authors are building?

L325: It seems a very strange choice to not use the same periods for each of the models being tested. I understand that there are computational costs to be accounted for, why not assess each model for the 1998-2001 period?

L600 Table 2. "Number of samples trained per iteration". Are the authors referring to batch size here? "Number of rounds to traverse the data set". Sorry, this is unclear to me. Is this stating that the training dataset contains 50 batches of 1024?

L620: "Note: Spatial averaging of MSE is performed before calculating ." This is unclear. Could the authors please explain further.

Figure 7: It would be very interesting to also plot the R2 values for the CAM5 parametrization as a model for the SP.

Figure 8: There appear to be negative R2 values in portions of the globe. This is a worryingly low skill for the model.

Figure 9: I think this figure could strongly benefit from a companion figure where the differences from the SPCAM run are shown for both CAM5 and NNCAM. Otherwise is it challenging to decipher if NNCAM lies closer to the SPCAM mean state than CAM5.

I also think it would be very interesting to compare all of these runs to the ERA5 state of the atmosphere for those years. This would go towards answering the question of whether SP is an improvement over CAM5.

Figure 10: As with figure 9, I think showing the differences would add significant information.

Figure 11: My interpretation of this plot is that NNCAM is a worse model of SPCAM than CAM5. Do the authors agree, and if so, why do they think this is true?

Figure 14: As with figure 11. It is not clear that NNCAM has succeeded at this task.
Citation: https://doi.org/10.5194/gmd-2021-299-RC2
- AC5: 'Reply on RC2', Xin Wang, 10 Jan 2022
  
  The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2021-299/gmd-2021-299-AC5-supplement.pdf
  
  Citation: https://doi.org/10.5194/gmd-2021-299-AC5

Peer review completion

AR: Author's response | RR: Referee report | ED: Editor decision | EF: Editorial file upload

AR by Xin Wang on behalf of the Authors (10 Jan 2022) Author's response Author's tracked changes Manuscript

EF by Polina Shvedko (10 Jan 2022) Supplement

ED: Reconsider after major revisions (02 Feb 2022) by Po-Lun Ma

AR by Xin Wang on behalf of the Authors (21 Mar 2022) Author's response Author's tracked changes Manuscript

EF by Sarah Buchmann (22 Mar 2022) Supplement

ED: Referee Nomination & Report Request started (28 Mar 2022) by Po-Lun Ma

RR by Anonymous Referee #1 (07 Apr 2022)

RR by Anonymous Referee #2 (11 Apr 2022)

ED: Publish subject to minor revisions (review by editor) (14 Apr 2022) by Po-Lun Ma

AR by Xin Wang on behalf of the Authors (19 Apr 2022) Author's response Author's tracked changes Manuscript

EF by Polina Shvedko (21 Apr 2022) Supplement

ED: Publish as is (22 Apr 2022) by Po-Lun Ma

AR by Xin Wang on behalf of the Authors (25 Apr 2022) Author's response Manuscript

Short summary

This study uses a set of deep neural networks to learn a parameterization scheme from a superparameterized general circulation model (GCM). After being embedded in a realistically configurated GCM, the parameterization scheme performs stably in long-term climate simulations and reproduces reasonable climatology and climate variability. This success is the first for long-term stable climate simulations using machine learning parameterization under real geographical boundary conditions.