The compression–error trade-off for large gridded data sets

Silver, Jeremy D.; Zender, Charles S.

doi:https://doi.org/10.5194/gmd-10-413-2017

Articles | Volume 10, issue 1

https://doi.org/10.5194/gmd-10-413-2017

© Author(s) 2017. This work is distributed under
the Creative Commons Attribution 3.0 License.

https://doi.org/10.5194/gmd-10-413-2017

© Author(s) 2017. This work is distributed under
the Creative Commons Attribution 3.0 License.

Articles | Volume 10, issue 1

Development and technical paper

|

27 Jan 2017

Development and technical paper |

| 27 Jan 2017

The compression–error trade-off for large gridded data sets

Jeremy D. Silver and Charles S. Zender

Download

Final revised paper (published on 27 Jan 2017)
Supplement to the final revised paper
Preprint (discussion started on 29 Jul 2016)

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

RC1: '"Finding the Goldilocks zone: Compression-error trade-off for large gridded datasets" by Jeremy D. Silver and Charles S. Zender', Anonymous Referee #1, 22 Aug 2016
- RC2: 'Review for layer-packing', Anonymous Referee #2, 30 Aug 2016
  - AC2: 'Reply to Reviewer 2, part 1', Jeremy David Silver, 28 Oct 2016
- RC3: 'Review responce', Anonymous Referee #2, 30 Aug 2016
  - AC3: 'Reply to reviewer 2, part 2', Jeremy David Silver, 28 Oct 2016
- AC1: 'Reply to referee 1', Jeremy David Silver, 28 Oct 2016
RC4: 'concerns about scientific significance', Anonymous Referee #3, 01 Sep 2016
- AC4: 'Reply to reviewer 3', Jeremy David Silver, 28 Oct 2016
AC5: 'Summary of changes, revised manuscript, complete reply to reviewers, new supplementary section', Jeremy David Silver, 28 Oct 2016

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

AR by Jeremy David Silver on behalf of the Authors (28 Oct 2016) Author's response Manuscript

ED: Referee Nomination & Report Request started (09 Nov 2016) by Paul Ullrich

RR by Anonymous Referee #3 (17 Nov 2016)

RR by Anonymous Referee #1 (18 Nov 2016)

ED: Publish as is (19 Nov 2016) by Paul Ullrich

AR by Jeremy David Silver on behalf of the Authors (06 Dec 2016) Author's response Manuscript

Short summary

Many modern scientific research projects generate large amounts of data. Storage space is valuable and may be limited; hence compression is vital. We tested different compression methods for large gridded data sets, assessing the space savings and the amount of precision lost. We found a general trade-off between precision and compression, with compression well-predicted by the entropy of the data set. A method introduced here proved to be a competitive archive format for gridded numerical data.