Submitted as: development and technical paper 12 Apr 2021

Submitted as: development and technical paper | 12 Apr 2021

Review status: this preprint is currently under review for the journal GMD.

Towards physically consistent data-driven weather forecasting: Integrating data assimilation with equivariance-preserving spatial transformers in a case study with ERA5

Ashesh Chattopadhyay1,2, Mustafa Mustafa2, Pedram Hassanzadeh1,3, Eviatar Bach4, and Karthik Kashinath2 Ashesh Chattopadhyay et al.
  • 1Department of Mechanical Engineering, Rice University, Houston, TX, USA
  • 2Lawrence Berkeley National Laboratory, Berkeley, CA, USA
  • 3Department of Earth, Environmental and Planetary Sciences, Rice University, Houston, TX, USA
  • 4Department of Atmospheric and Oceanic Science and Institute for Physical Science and Technology, University of Maryland, College Park, USA

Abstract. There is growing interest in data-driven weather prediction (DDWP), for example using convolutional neural networks such as U-NETs that are trained on data from models or reanalysis. Here, we propose 3 components to integrate with commonly used DDWP models in order to improve their physical consistency and forecast accuracy. These components are 1) a deep spatial transformer added to the latent space of the U-NETs to preserve a property called equivariance, which is related to correctly capturing rotations and scalings of features in spatio-temporal data, 2) a data-assimilation (DA) algorithm to ingest noisy observations and improve the initial conditions for next forecasts, and 3) a multi-time-step algorithm, which combines forecasts from DDWP models with different time steps through DA, improving the accuracy of forecasts at short intervals. To show the benefit/feasibility of each component, we use geopotential height at 500 hPa (Z500) from ERA5 reanalysis and examine the short-term forecast accuracy of specific setups of the DDWP framework. Results show that the equivariance-preserving networks (U-STNs) clearly outperform the U-NETs, for example improving the forecast skill by 45 %. Using a sigma-point ensemble Kalman (SPEnKF) algorithm for DA and U-STN as the forward model, we show that stable, accurate DA cycles are achieved even with high observation noise. The DDWP+DA framework substantially benefits from large (O(1000)) ensembles that are inexpensively generated with the data-driven forward model in each DA cycle. The multi-time-step DDWP+DA framework also shows promises, e.g., it reduces the average error by factors of 2–3. These results show the benefits/feasibilities of these 3 components, which are flexible and can be used in a variety of DDWP setups. Furthermore, while here we focus on weather forecasting, the 3 components can be readily adopted for other parts of the Earth system, such as ocean and land, for which there is a rapid growth of data and need for forecast/assimilation.

Ashesh Chattopadhyay et al.

Status: final response (author comments only)

Comment types: AC – author | RC – referee | CC – community | EC – editor | CEC – chief editor | : Report abuse
  • RC1: 'Referee comment on gmd-2021-71', Anonymous Referee #1, 31 May 2021
    • AC1: 'Reply on RC1', Ashesh Chattopadhyay, 23 Aug 2021
  • RC2: 'Comment on gmd-2021-71', Anonymous Referee #2, 02 Jun 2021
    • AC2: 'Reply on RC2', Ashesh Chattopadhyay, 23 Aug 2021

Ashesh Chattopadhyay et al.

Ashesh Chattopadhyay et al.


Total article views: 1,087 (including HTML, PDF, and XML)
HTML PDF XML Total BibTeX EndNote
486 584 17 1,087 17 13
  • HTML: 486
  • PDF: 584
  • XML: 17
  • Total: 1,087
  • BibTeX: 17
  • EndNote: 13
Views and downloads (calculated since 12 Apr 2021)
Cumulative views and downloads (calculated since 12 Apr 2021)

Viewed (geographical distribution)

Total article views: 987 (including HTML, PDF, and XML) Thereof 987 with geography defined and 0 with unknown origin.
Country # Views %
  • 1


Latest update: 27 Nov 2021
Short summary
There is growing interest in data-driven weather forecasting, i.e., to predict the weather by using a deep neural network that learns from the evolution of past atmospheric patterns. Here, we propose 3 components to add to the current data-driven weather forecast models to improve their performance. These components involve a feature that incorporates physics into the neural network, a method to add data assimilation, and an algorithm to use several different time intervals in the forecast.