Articles | Volume 12, issue 9
Geosci. Model Dev., 12, 4099–4113, 2019
https://doi.org/10.5194/gmd-12-4099-2019
Geosci. Model Dev., 12, 4099–4113, 2019
https://doi.org/10.5194/gmd-12-4099-2019

Development and technical paper 23 Sep 2019

Development and technical paper | 23 Sep 2019

Evaluation of lossless and lossy algorithms for the compression of scientific datasets in netCDF-4 or HDF5 files

Xavier Delaunay et al.

Related authors

A Parquet Cube alternative to store gridded data for data analytics and modeling
Jean-Michel Zigna, Reda Semlal, Flavien Gouillon, Ethan Davis, Elisabeth Lambert, Frédéric Briol, Romain Prod-Homme, Sean Arms, and Lionel Zawadzki
Geosci. Model Dev. Discuss., https://doi.org/10.5194/gmd-2021-138,https://doi.org/10.5194/gmd-2021-138, 2021
Revised manuscript under review for GMD
Short summary

Related subject area

Numerical methods
B-flood 1.0: an open-source Saint-Venant model for flash-flood simulation using adaptive refinement
Geoffroy Kirstetter, Olivier Delestre, Pierre-Yves Lagrée, Stéphane Popinet, and Christophe Josserand
Geosci. Model Dev., 14, 7117–7132, https://doi.org/10.5194/gmd-14-7117-2021,https://doi.org/10.5194/gmd-14-7117-2021, 2021
Short summary
A micro-genetic algorithm (GA v1.7.1a) for combinatorial optimization of physics parameterizations in the Weather Research and Forecasting model (v4.0.3) for quantitative precipitation forecast in Korea
Sojung Park and Seon K. Park
Geosci. Model Dev., 14, 6241–6255, https://doi.org/10.5194/gmd-14-6241-2021,https://doi.org/10.5194/gmd-14-6241-2021, 2021
Short summary
SymPKF (v1.0): a symbolic and computational toolbox for the design of parametric Kalman filter dynamics
Olivier Pannekoucke and Philippe Arbogast
Geosci. Model Dev., 14, 5957–5976, https://doi.org/10.5194/gmd-14-5957-2021,https://doi.org/10.5194/gmd-14-5957-2021, 2021
Short summary
NDCmitiQ v1.0.0: a tool to quantify and analyse greenhouse gas mitigation targets
Annika Günther, Johannes Gütschow, and Mairi Louise Jeffery
Geosci. Model Dev., 14, 5695–5730, https://doi.org/10.5194/gmd-14-5695-2021,https://doi.org/10.5194/gmd-14-5695-2021, 2021
Short summary
Combining ensemble Kalman filter and reservoir computing to predict spatiotemporal chaotic systems from imperfect observations and models
Futo Tomizawa and Yohei Sawada
Geosci. Model Dev., 14, 5623–5635, https://doi.org/10.5194/gmd-14-5623-2021,https://doi.org/10.5194/gmd-14-5623-2021, 2021
Short summary

Cited articles

Baker, A. H., Hammerling, D. M., Mickelson, S. A., Xu, H., Stolpe, M. B., Naveau, P., Sanderson, B., Ebert-Uphoff, I., Samarasinghe, S., De Simone, F., Carbone, F., Gencarelli, C. N., Dennis, J. M., Kay, J. E., and Lindstrom, P.: Evaluating lossy data compression on climate simulation data within a large ensemble, Geosci. Model Dev., 9, 4381–4403, https://doi.org/10.5194/gmd-9-4381-2016, 2016. 
Caron, J.: Compression by Scaling and Offset, available at: http://www.unidata.ucar.edu/blogs/developer/en/entry/compression_by_scaling_and_offfset (last access: 27 September 2018), 2014a. 
Caron, J.: Compression by bit shaving, available at: http://www.unidata.ucar.edu/blogs/developer/entry/compression_by_bit_shaving (last access: 27 September 2018), 2014b. 
Collet, Y.: LZ4 lossless compression algorithm, available at: http://lz4.org (last access: 27 September 2018), 2013. 
Collet, Y. and Turner, C.: Smaller and faster data compression with Zstandard, available at: https://code.fb.com/core-data/smaller-and-faster-data-compression-with-zstandard/ (last access: 27 September 2018), 2016. 
Download
Short summary
This research aimed at finding a compression method suitable for the ground processing of CFOSAT and SWOT satellite datasets. Lossless algorithms did not allow enough compression. That is why we began studying lossy alternatives. This work introduces the digit rounding algorithm which reduces the volume of scientific datasets keeping only the significant digits in each sample value. The number of digits kept is relative to each sample so that both small and high values are similarly preserved.