Efficient high-dimensional variational data assimilation with machine-learned reduced-order models

Maulik, Romit; Rao, Vishwas; Wang, Jiali; Mengaldo, Gianmarco; Constantinescu, Emil; Lusch, Bethany; Balaprakash, Prasanna; Foster, Ian; Kotamarthi, Rao

doi:https://doi.org/10.5194/gmd-15-3433-2022

Articles | Volume 15, issue 8

https://doi.org/10.5194/gmd-15-3433-2022

© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/gmd-15-3433-2022

© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 15, issue 8

Development and technical paper

|

02 May 2022

Development and technical paper |

| 02 May 2022

Efficient high-dimensional variational data assimilation with machine-learned reduced-order models

Romit Maulik, Vishwas Rao, Jiali Wang, Gianmarco Mengaldo, Emil Constantinescu, Bethany Lusch, Prasanna Balaprakash, Ian Foster, and Rao Kotamarthi

Download

Final revised paper (published on 02 May 2022)
Preprint (discussion started on 07 Jan 2022)

Interactive discussion

Status: closed

RC1:
'Comment on gmd-2021-415', Anonymous Referee #1, 18 Feb 2022

The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2021-415/gmd-2021-415-RC1-supplement.pdf

Citation: https://doi.org/10.5194/gmd-2021-415-RC1
- AC2: 'Reply on RC1', Romit Maulik, 14 Mar 2022
  
  Thank you for your comments. We have addressed them in the revision. We also attach a response letter here.
  
  Citation: https://doi.org/10.5194/gmd-2021-415-AC2
CEC1:
'Comment on gmd-2021-415', Juan Antonio Añel, 22 Feb 2022

After checking your manuscript, it has come to our attention that it does not comply with our Code and Data Policy.

https://www.geoscientific-model-development.net/policies/code_and_data_policy.html
We can not accept embargoes such as registration or previous contact with the authors to get access to data or code. Therefore, please, to be able of considering your paper for publication you must publish the code and data that you have used for your work in one of the appropriate repositories according to our policy.

In this way, you must reply to this comment with the link to the repository used in your manuscript, with its DOI. We understand that some files used in your study can be large (e.g., full output from models). In such cases, instead of storing the complete files, you should at least keep the variables or final fields computed and used in your work.
Also, you must include in a potential reviewed version of your manuscript the modified 'Code and Data Availability' section and the DOI of the code and data repository.
Please, when publishing the code, be aware that If you do not include a license, the code continues to be your property and can not be used by others, despite any statement on being free to use. Therefore, when uploading code, you could want to choose a free software/open-source (FLOSS) license. We recommend the GPLv3. You only need to include the file 'https://www.gnu.org/licenses/gpl-3.0.txt' as LICENSE.txt with your code. Also, you can choose other options that Zenodo provides: GPLv2, Apache License, MIT License, etc.
Please, reply as soon as possible to this comment with the link for it so that it is available for the peer-review process, as it should be.
Juan A. Añel

Geosci. Model Dev. Exec. Editor

Citation: https://doi.org/10.5194/gmd-2021-415-CEC1
- AC1: 'Reply on CEC1', Romit Maulik, 28 Feb 2022
  
  Dear Dr. Añel - thank you for your comment. Our repository and data set is now public and available here:
  https://github.com/AIEADA/LSTM_Var_Prototype
  We are continuing to work on the revision to the Referee comments and will update the main manuscript shortly.
  
  Citation: https://doi.org/10.5194/gmd-2021-415-AC1
RC2:
'Comment on gmd-2021-415', Anonymous Referee #2, 09 Mar 2022

Excellent paper. Great improvement from previous studies where the ML approach learns only the misfits whilst the authors have implemented an on-the-fly system.

Figure 3 and 4 need to be together for a better comparison.

Architectures of networks used?

Was layer normalisation used? Why not?

What other temporal NN could be used? Transformers?

What is the effort of using a CNN instead of a PCA reduction to feed the LSTM AE?

I would like to see a brief discussion of how an adversarial approach might help the rollout of the forecast here.

Citation: https://doi.org/10.5194/gmd-2021-415-RC2
- AC3: 'Reply on RC2', Romit Maulik, 14 Mar 2022
  
  Thank you for your comments. We have addressed them in our revision. We also attach a response letter here.
  
  Citation: https://doi.org/10.5194/gmd-2021-415-AC3

Peer review completion

AR: Author's response | RR: Referee report | ED: Editor decision | EF: Editorial file upload

AR by Romit Maulik on behalf of the Authors (14 Mar 2022) Author's response Author's tracked changes Manuscript

ED: Publish subject to minor revisions (review by editor) (23 Mar 2022) by Xiaomeng Huang

AR by Romit Maulik on behalf of the Authors (24 Mar 2022) Author's response Author's tracked changes Manuscript

ED: Publish as is (26 Mar 2022) by Xiaomeng Huang

AR by Romit Maulik on behalf of the Authors (29 Mar 2022) Manuscript

Download

The requested paper has a corresponding corrigendum published. Please read the corrigendum first before downloading the article.

Article (2884 KB)
Full-text XML

Short summary

In numerical weather prediction, data assimilation is frequently utilized to enhance the accuracy of forecasts from equation-based models. In this work we use a machine learning framework that approximates a complex dynamical system given by the geopotential height. Instead of using an equation-based model, we utilize this machine-learned alternative to dramatically accelerate both the forecast and the assimilation of data, thereby reducing need for large computational resources.