Preprints
https://doi.org/10.5194/gmd-2021-65
https://doi.org/10.5194/gmd-2021-65

Submitted as: development and technical paper 21 Jun 2021

Submitted as: development and technical paper | 21 Jun 2021

Review status: this preprint is currently under review for the journal GMD.

Evaluation and optimisation of the I/O scalability for the next generation of Earth system models: IFS CY43R3 and XIOS 2.0 integration as a case study

Xavier Yepes-Arbós1, Gijs van den Oord2, Mario C. Acosta1, and Glenn D. Carver3 Xavier Yepes-Arbós et al.
  • 1Barcelona Supercomputing Center - Centro Nacional de Supercomputación (BSC-CNS), Barcelona, Spain
  • 2Netherlands eScience Center (NLeSC), Amsterdam, The Netherlands
  • 3European Centre for Medium-Range Weather Forecasts (ECMWF), Reading, United Kingdom

Abstract. Earth system models have considerably increased their spatial resolution to solve more complex problems and achieve more realistic solutions. However, this generates an enormous amount of model data which requires proper management. Some Earth system models use inefficient sequential Input/Output (I/O) schemes that do not scale well when many parallel resources are used. In order to address this issue, the most commonly adopted approach is to use scalable parallel I/O solutions that offer both computational performance and efficiency.

In this paper we analyse the I/O process of the European Centre for Medium-Range Weather Forecasts (ECMWF) operational Integrated Forecasting System (IFS) CY43R3. IFS can use two different output schemes: a parallel I/O server developed by MeteoFrance used operationally, and an obsolescent sequential I/O scheme. The latter is the only scheme that is being exposed by the OpenIFS variant of IFS. “Downstream” Earth system models that have adopted older versions of an IFS derivative as a component – such as the EC-Earth 3 climate model – also face a bottleneck due to the limited I/O capabilities and performance of the sequential output scheme. Moreover it is often desirable to produce gridpoint-space Network Common Data Format (NetCDF) files instead of the IFS native spectral and gridpoint output fields in General Regularly-distributed Information in Binary form (GRIB) format, which requires the development of model-specific post-processing tools.

We present the integration of the XML Input/Output Server (XIOS) 2.0 into IFS CY43R3. XIOS is an asynchronous Message Passing Interface (MPI) I/O server that offers features especially targeted at climate models: NetCDF format output files, inline diagnostics, regridding, and when properly configured, the capability to produce CMOR-compliant data. We therefore expect our work to reduce the computational cost of data-intensive (high-resolution) climate runs, thereby shortening the critical path of EC-Earth 4 experiments.

The performance evaluation suggests that the use of XIOS 2.0 in IFS CY43R3 to output data achieves an adequate performance as well outperforming the sequential I/O scheme. Furthermore, when we also take into account the post-processing task, needed to convert GRIB files to NetCDF files, and also transform IFS spectral output fields to gridpoint space, our integration not only surpasses the sequential output scheme but also the IFS I/O server.

Xavier Yepes-Arbós et al.

Status: open (until 28 Aug 2021)

Comment types: AC – author | RC – referee | CC – community | EC – editor | CEC – chief editor | : Report abuse

Xavier Yepes-Arbós et al.

Data sets

Evaluation and optimisation of the I/O scalability for the next generation of Earth system models: IFS CY43R3 and XIOS 2.0 integration as a case study Yepes-Arbós, Xavier, van den Oord, Gijs, Acosta, Mario C., & Carver, Glenn https://doi.org/10.5281/zenodo.4473008

Model code and software

IFS CY43R3 and XIOS 2.0 integration Yepes-Arbós, Xavier, & van den Oord, Gijs https://doi.org/10.5281/zenodo.4905832

XIOS 2.0 (Revision 1297) Meurdesoif, Yann https://doi.org/10.5281/zenodo.4905653

XIOS-GRIB compare van den Oord, Gijs https://doi.org/10.5281/zenodo.4906175

Xavier Yepes-Arbós et al.

Viewed

Total article views: 317 (including HTML, PDF, and XML)
HTML PDF XML Total BibTeX EndNote
253 56 8 317 7 4
  • HTML: 253
  • PDF: 56
  • XML: 8
  • Total: 317
  • BibTeX: 7
  • EndNote: 4
Views and downloads (calculated since 21 Jun 2021)
Cumulative views and downloads (calculated since 21 Jun 2021)

Viewed (geographical distribution)

Total article views: 290 (including HTML, PDF, and XML) Thereof 290 with geography defined and 0 with unknown origin.
Country # Views %
  • 1
1
 
 
 
 
Latest update: 30 Jul 2021
Download
Short summary
Climate prediction models produce a large volume of simulated data that sometimes might not be efficiently managed. In this paper we present an approach to address this issue by reducing the computing time and storage space. As a case study, we analyse the output writing process of the ECMWF atmospheric model called IFS, and we integrate into it a data writing tool called XIOS. The results suggest that the integration between both components achieves an adequate computational performance.