Articles | Volume 9, issue 9
Geosci. Model Dev., 9, 3199–3211, 2016
https://doi.org/10.5194/gmd-9-3199-2016
© Author(s) 2016. This work is distributed under
the Creative Commons Attribution 3.0 License.
the Creative Commons Attribution 3.0 License.
Geosci. Model Dev., 9, 3199–3211, 2016
https://doi.org/10.5194/gmd-9-3199-2016
© Author(s) 2016. This work is distributed under
the Creative Commons Attribution 3.0 License.
the Creative Commons Attribution 3.0 License.
Development and technical paper
19 Sep 2016
Development and technical paper
| 19 Sep 2016
Bit Grooming: statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+)
Charles S. Zender
Viewed
Total article views: 3,179 (including HTML, PDF, and XML)
Cumulative views and downloads
(calculated since 21 Apr 2016)
| HTML | XML | Total | Supplement | BibTeX | EndNote | |
|---|---|---|---|---|---|---|
| 2,021 | 997 | 161 | 3,179 | 519 | 167 | 165 |
- HTML: 2,021
- PDF: 997
- XML: 161
- Total: 3,179
- Supplement: 519
- BibTeX: 167
- EndNote: 165
Total article views: 2,693 (including HTML, PDF, and XML)
Cumulative views and downloads
(calculated since 19 Sep 2016)
| HTML | XML | Total | Supplement | BibTeX | EndNote | |
|---|---|---|---|---|---|---|
| 1,759 | 783 | 151 | 2,693 | 392 | 155 | 156 |
- HTML: 1,759
- PDF: 783
- XML: 151
- Total: 2,693
- Supplement: 392
- BibTeX: 155
- EndNote: 156
Total article views: 486 (including HTML, PDF, and XML)
Cumulative views and downloads
(calculated since 21 Apr 2016)
| HTML | XML | Total | Supplement | BibTeX | EndNote | |
|---|---|---|---|---|---|---|
| 262 | 214 | 10 | 486 | 127 | 12 | 9 |
- HTML: 262
- PDF: 214
- XML: 10
- Total: 486
- Supplement: 127
- BibTeX: 12
- EndNote: 9
Cited
13 citations as recorded by crossref.
- The compression–error trade-off for large gridded data sets J. Silver & C. Zender 10.5194/gmd-10-413-2017
- Advancing data compression via noise detection D. Hammerling & A. Baker 10.1038/s43588-021-00167-z
- Evaluation of lossless and lossy algorithms for the compression of scientific datasets in netCDF-4 or HDF5 files X. Delaunay et al. 10.5194/gmd-12-4099-2019
- Compressing atmospheric data into its real information content M. Klöwer et al. 10.1038/s43588-021-00156-2
- Parallel Implementation of a PETSc-Based Framework for the General Curvilinear Coastal Ocean Model M. Valera et al. 10.3390/jmse7060185
- Array DBMS R. Zalipynis 10.14778/3476311.3476404
- New Methods for Data Storage of Model Output from Ensemble Simulations P. Düben et al. 10.1175/MWR-D-18-0170.1
- Evaluating image quality measures to assess the impact of lossy data compression applied to climate simulation data A. Baker et al. 10.1111/cgf.13707
- Data-Driven Artificial Intelligence for Calibration of Hyperspectral Big Data V. Sagan et al. 10.1109/TGRS.2021.3091409
- A statistical analysis of lossily compressed climate model data A. Poppick et al. 10.1016/j.cageo.2020.104599
- GenomicScores: seamless access to genomewide position-specific scores from R and Bioconductor P. Puigdevall et al. 10.1093/bioinformatics/bty311
- Evaluating lossy data compression on climate simulation data within a large ensemble A. Baker et al. 10.5194/gmd-9-4381-2016
- A note on precision-preserving compression of scientific data R. Kouznetsov 10.5194/gmd-14-377-2021
13 citations as recorded by crossref.
- The compression–error trade-off for large gridded data sets J. Silver & C. Zender 10.5194/gmd-10-413-2017
- Advancing data compression via noise detection D. Hammerling & A. Baker 10.1038/s43588-021-00167-z
- Evaluation of lossless and lossy algorithms for the compression of scientific datasets in netCDF-4 or HDF5 files X. Delaunay et al. 10.5194/gmd-12-4099-2019
- Compressing atmospheric data into its real information content M. Klöwer et al. 10.1038/s43588-021-00156-2
- Parallel Implementation of a PETSc-Based Framework for the General Curvilinear Coastal Ocean Model M. Valera et al. 10.3390/jmse7060185
- Array DBMS R. Zalipynis 10.14778/3476311.3476404
- New Methods for Data Storage of Model Output from Ensemble Simulations P. Düben et al. 10.1175/MWR-D-18-0170.1
- Evaluating image quality measures to assess the impact of lossy data compression applied to climate simulation data A. Baker et al. 10.1111/cgf.13707
- Data-Driven Artificial Intelligence for Calibration of Hyperspectral Big Data V. Sagan et al. 10.1109/TGRS.2021.3091409
- A statistical analysis of lossily compressed climate model data A. Poppick et al. 10.1016/j.cageo.2020.104599
- GenomicScores: seamless access to genomewide position-specific scores from R and Bioconductor P. Puigdevall et al. 10.1093/bioinformatics/bty311
- Evaluating lossy data compression on climate simulation data within a large ensemble A. Baker et al. 10.5194/gmd-9-4381-2016
- A note on precision-preserving compression of scientific data R. Kouznetsov 10.5194/gmd-14-377-2021
Saved (final revised paper)
Latest update: 30 Jan 2023
Short summary
We introduce Bit Grooming, a lossy compression algorithm that removes the bloat due to false precision, those bits and bytes beyond the meaningful precision of the data. Bit Grooming is statistically unbiased, applies to all floating-point numbers, and is easy to use. Bit Grooming reduces data storage requirements by 25–80 %. Unlike its best-known competitor Linear Packing, Bit Grooming imposes no software overhead on users, and guarantees its precision throughout the whole floating-point range.
We introduce Bit Grooming, a lossy compression algorithm that removes the bloat due to false...
