Evaluation of lossless and lossy algorithms for the compression of scientific datasets in netCDF-4 or HDF5 files

Delaunay, Xavier; Courtois, Aurélie; Gouillon, Flavien

doi:https://doi.org/10.5194/gmd-12-4099-2019

Articles | Volume 12, issue 9

https://doi.org/10.5194/gmd-12-4099-2019

© Author(s) 2019. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/gmd-12-4099-2019

© Author(s) 2019. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 12, issue 9

Development and technical paper

|

23 Sep 2019

Development and technical paper |

| 23 Sep 2019

Evaluation of lossless and lossy algorithms for the compression of scientific datasets in netCDF-4 or HDF5 files

Xavier Delaunay, Aurélie Courtois, and Flavien Gouillon

Download

Final revised paper (published on 23 Sep 2019)
Supplement to the final revised paper
Preprint (discussion started on 20 Nov 2018)

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

RC1: 'Review for Delaunay et al. "Evaluation of lossless and lossy compression ..."', Anonymous Referee #1, 16 Jan 2019
- AC1: 'Response to referee#1', Xavier Delaunay, 20 Feb 2019
RC2: 'Review of Evaluation of lossless and lossy algorithms for the compression of scientific datasets in NetCDF-4 or HDF5 formatted files', Charles Zender, 26 Jan 2019
- AC2: 'Response to referee C. Zender', Xavier Delaunay, 20 Feb 2019

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

AR by Anna Mirena Feist-Polner on behalf of the Authors (04 Apr 2019) Author's response

ED: Referee Nomination & Report Request started (15 May 2019) by Steve Easterbrook

RR by Anonymous Referee #3 (20 May 2019)

Suggestions for revision or reasons for rejection

The paper studied the performance of a series of lossless and lossy compression methods/filters that can be plugged in NetCDF-4/HDF5 data for scientific floating-point datasets. They also proposed a Digital Rounding algorithm and compare it with Sz and Bit Grooming algorithm on two synthetic datasets and two real-world datasets.

Comments:
- Introduction: as mentioned in this section, FPZIP and ZFP are efficient and effective lossy compressors. Is there any reason why didn't evaluate them? I noticed that referee 1 also asked this same question. Actually, FPZIP is not only lossless but also lossy compression with precision mode. I suggest testing ZFP/FPZIP or briefly explain why FPZIP/ZFP are opted out (e.g., worse than Sz).
- Compression algorithms: the difference of Decimal Rounding and Digit Rounding algorithm seems blurred to me. Can you explain more details about the Decimal Rounding algorithm?
- Application: is there any reason/motivation to compress CFOSAT and SWOT datasets? It's better to explicitly describe the (projected) data volumes that will be produced by these missions.
- Conclusion: This statement seems unconvincing to me: "Sz provides higher compression ratios than Decimal Rounding on most datasets. However, for the compression of real scientific datasets, its usability is reduced by the fact that only one error bound can be set for all the variables composing the dataset. It cannot be easily configured to achieve the precision required variable per variable." Actually, Sz provides value-range-based relative error bound, it can apply different absolute error bounds to different variables in the dataset based on the value range of each variable. Please explicitly explain if this mode can satisfy your demand.
- There are still some grammatical issues that need to be fixed, e.g., "Sz provide" -> "Sz provides", "the compressions obtained" -> "the compression ratios obtained", etc.
- The major concern of this paper is the generality and novelty of Digital Rounding algorithm. Based on the experiments, it looks Digital Rounding algorithm is comparatively as good as other approaches and better in some aspects, but the evaluation is only conducted on two datasets/applications. Also, this algorithm looks very similar to Sz's linearly scaling quantization. If not, the paper should explicitly explain the difference.

Hide

RR by Anonymous Referee #1 (27 May 2019)

Suggestions for revision or reasons for rejection

The authors clearly put a lot of effort into this revision, and the quality of the manuscript has greatly improved. I am overall satisfied with their responses to my comments and suggestions. In particular, the additional details on the algorithms and test data are much appreciated as is the improvement in focus. The contributions have also been clarified. Finally, I'll note that the writing quality is very much improved for this revision. For subsequent manuscripts, I would encourage the authors to aim for this higher quality of writing in the initial submission.

A few minor issues:

p.3, line 13: "Both algorithms..." - which two are being referred to? Or is it to all three (in which case, it should by "All three algorithms...")

p.3, line 29: "... to degrade the least significant ..." This use of "degrade" is a bit awkward - consider rephrasing - maybe "to remove the least significant"

p.4, line 29: "consists in" => "consists of"

p. 5: Section 3 (line 16) refers to Table 3. Note that Table 3 lists mean error, mean absolute error, and maximum absolute error. However, these metrics are not defined until Section 4. Should section 3 and 4 be switched? Or a forward reference added?

p.5: Section 3 also refers to Table 4, which uses CR -- which is not defined until Section 4. Though it seems that in Table 4, CR is a percentage as opposed to a ratio
as defined in Section 4, so this needs to be clarified.

p.5, line 19: "...a relative error ..." I think you mean "absolute error"

p.10, line 10: "compressions" => "compression ratios"

p. 13, line 27: "was configured: => "were configured"

p.13, line 34: "provides" => "provide"

p. 14, line 23: "fails achieving" => "fails to achieve"

Hide

ED: Publish subject to minor revisions (review by editor) (28 May 2019) by Steve Easterbrook

AR by Xavier Delaunay on behalf of the Authors (07 Jun 2019) Author's response Manuscript

ED: Publish as is (03 Jul 2019) by Steve Easterbrook

Short summary

This research aimed at finding a compression method suitable for the ground processing of CFOSAT and SWOT satellite datasets. Lossless algorithms did not allow enough compression. That is why we began studying lossy alternatives. This work introduces the digit rounding algorithm which reduces the volume of scientific datasets keeping only the significant digits in each sample value. The number of digits kept is relative to each sample so that both small and high values are similarly preserved.