Preprints
https://doi.org/10.5194/gmd-2020-124
https://doi.org/10.5194/gmd-2020-124

Submitted as: development and technical paper 02 Jun 2020

Submitted as: development and technical paper | 02 Jun 2020

Review status: a revised version of this preprint was accepted for the journal GMD and is expected to appear here in due course.

Adaptive lossy compression of climate model data based on hierarchical tensor with Adaptive-HGFDR (v1.0)

Zhaoyuan Yu1,2, Zhengfang Zhang1, Dongshuang Li3,4, Wen Luo1,2, Yuan Liu1, Uzair Bhatti1, and Linwang Yuan1,2 Zhaoyuan Yu et al.
  • 1Key Laboratory of Virtual Geographic Environment, Ministry of Education, Nanjing Normal University, Nanjing, China
  • 2Jiangsu Center for Collaborative Innovation in Geographical Information Resource Development and Application, Nanjing, China
  • 3Jiangsu Key Laboratory of Crop Genetics and Physiology/Jiangsu Key Laboratory of Crop Cultivation and Physiology, Agricultural College of Yangzhou University, Yangzhou, China
  • 4Jiangsu Co Innovation Center for Modern Production Technology of Grain Crops, Yangzhou University, Yangzhou, China

Abstract. Lossy compression has been applied to large-scale experimental model data compression due to its advantages of a high compression ratio. However, few methods consider the uneven distribution of compression errors affecting compression quality. Here we develop an adaptive lossy compression method with the stable compression error for earth system model data based on Hierarchical Geospatial Field Data Representation (HGFDR). We extended the original HGFDR by firstly dividing the original data into a series of the local block according to the exploratory experiment to maximize the local correlations of the data. After that, from the mathematical model of the HGFDR, the relationship between the compression parameter and compression error in HGFDR for each block is analyzed and calculated. Using optimal compression parameter selection rule and an adaptive compression algorithm, our method, the Adaptive-HGFDR, achieved the data compression under the constraints that the compression error is as stable as possible through each dimension. Experiments concerning model data compression are carried out based on the Community Earth System Model (CESM) data. The results show that our method has higher compression ratio and more uniform error distributions, compared with other commonly used lossy compression methods, such as the Fixed-Rate Compressed Floating-Point Arrays method.

Zhaoyuan Yu et al.

 
Status: closed
Status: closed
AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment
Printer-friendly Version - Printer-friendly version Supplement - Supplement
 
Status: closed
Status: closed
AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment
Printer-friendly Version - Printer-friendly version Supplement - Supplement

Zhaoyuan Yu et al.

Data sets

AndyWZJ/Adaptive-lossy-compression- v1.0 W. Z. J. Andy https://doi.org/10.5281/zenodo.3862130

Model code and software

AndyWZJ/Adaptive-lossy-compression- v1.0 W. Z. J. Andy https://doi.org/10.5281/zenodo.3862130

Executable research compendia (ERC)

AndyWZJ/Adaptive-lossy-compression- v1.0 W. Z. J. Andy https://doi.org/10.5281/zenodo.3862130

Zhaoyuan Yu et al.

Viewed

Total article views: 350 (including HTML, PDF, and XML)
HTML PDF XML Total BibTeX EndNote
222 92 36 350 35 37
  • HTML: 222
  • PDF: 92
  • XML: 36
  • Total: 350
  • BibTeX: 35
  • EndNote: 37
Views and downloads (calculated since 02 Jun 2020)
Cumulative views and downloads (calculated since 02 Jun 2020)

Viewed (geographical distribution)

Total article views: 245 (including HTML, PDF, and XML) Thereof 245 with geography defined and 0 with unknown origin.
Country # Views %
  • 1
1
 
 
 
 
Latest update: 26 Jan 2021
Download
Short summary
Few methods consider the uneven distribution of compression errors affecting compression quality. Here we develop an adaptive lossy compression method with the stable compression error for earth system model data. The results show that our method has higher compression ratio and more uniform error distributions, compared with other commonly used lossy compression methods, such as the Fixed-Rate Compressed Floating-Point Arrays method.