Articles | Volume 9, issue 9
https://doi.org/10.5194/gmd-9-3199-2016
© Author(s) 2016. This work is distributed under
the Creative Commons Attribution 3.0 License.
the Creative Commons Attribution 3.0 License.
https://doi.org/10.5194/gmd-9-3199-2016
© Author(s) 2016. This work is distributed under
the Creative Commons Attribution 3.0 License.
the Creative Commons Attribution 3.0 License.
Development and technical paper
| 
19 Sep 2016
Development and technical paper |
| 19 Sep 2016
Bit Grooming: statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+)
Departments of Earth System Science and Computer Science, University of California, Irvine, Irvine, CA 92697-3100, USA
Viewed
Total article views: 4,547 (including HTML, PDF, and XML)
Cumulative views and downloads
(calculated since 21 Apr 2016)
| HTML | XML | Total | Supplement | BibTeX | EndNote | |
|---|---|---|---|---|---|---|
| 3,029 | 1,293 | 225 | 4,547 | 636 | 235 | 220 |
- HTML: 3,029
- PDF: 1,293
- XML: 225
- Total: 4,547
- Supplement: 636
- BibTeX: 235
- EndNote: 220
Total article views: 3,983 (including HTML, PDF, and XML)
Cumulative views and downloads
(calculated since 19 Sep 2016)
| HTML | XML | Total | Supplement | BibTeX | EndNote | |
|---|---|---|---|---|---|---|
| 2,762 | 1,007 | 214 | 3,983 | 485 | 223 | 211 |
- HTML: 2,762
- PDF: 1,007
- XML: 214
- Total: 3,983
- Supplement: 485
- BibTeX: 223
- EndNote: 211
Total article views: 564 (including HTML, PDF, and XML)
Cumulative views and downloads
(calculated since 21 Apr 2016)
| HTML | XML | Total | Supplement | BibTeX | EndNote | |
|---|---|---|---|---|---|---|
| 267 | 286 | 11 | 564 | 151 | 12 | 9 |
- HTML: 267
- PDF: 286
- XML: 11
- Total: 564
- Supplement: 151
- BibTeX: 12
- EndNote: 9
Cited
19 citations as recorded by crossref.
- Advancing data compression via noise detection D. Hammerling & A. Baker 10.1038/s43588-021-00167-z
- Discussion on “Saving Storage in Climate Ensembles: A Model-Based Stochastic Approach” J. Bessac et al. 10.1007/s13253-023-00540-7
- Black-box statistical prediction of lossy compression ratios for scientific data R. Underwood et al. 10.1177/10943420231179417
- The impact of altering emission data precision on compression efficiency and accuracy of simulations of the community multiscale air quality model M. Walters & D. Wong 10.5194/gmd-16-1179-2023
- Compressing atmospheric data into its real information content M. Klöwer et al. 10.1038/s43588-021-00156-2
- Array DBMS R. Zalipynis 10.14778/3476311.3476404
- Optimizing Error-Bounded Lossy Compression for Scientific Data With Diverse Constraints Y. Liu et al. 10.1109/TPDS.2022.3194695
- The compression–error trade-off for large gridded data sets J. Silver & C. Zender 10.5194/gmd-10-413-2017
- Improving Performance of SLAV Model for Medium Range Weather Prediction R. Fadeev et al. 10.1134/S1995080224603874
- Evaluation of lossless and lossy algorithms for the compression of scientific datasets in netCDF-4 or HDF5 files X. Delaunay et al. 10.5194/gmd-12-4099-2019
- Parallel Implementation of a PETSc-Based Framework for the General Curvilinear Coastal Ocean Model M. Valera et al. 10.3390/jmse7060185
- New Methods for Data Storage of Model Output from Ensemble Simulations P. Düben et al. 10.1175/MWR-D-18-0170.1
- Evaluating image quality measures to assess the impact of lossy data compression applied to climate simulation data A. Baker et al. 10.1111/cgf.13707
- Data-Driven Artificial Intelligence for Calibration of Hyperspectral Big Data V. Sagan et al. 10.1109/TGRS.2021.3091409
- A statistical analysis of lossily compressed climate model data A. Poppick et al. 10.1016/j.cageo.2020.104599
- GenomicScores: seamless access to genomewide position-specific scores from R and Bioconductor P. Puigdevall et al. 10.1093/bioinformatics/bty311
- Evaluating lossy data compression on climate simulation data within a large ensemble A. Baker et al. 10.5194/gmd-9-4381-2016
- Multifacets of lossy compression for scientific data in the Joint-Laboratory of Extreme Scale Computing F. Cappello et al. 10.1016/j.future.2024.05.022
- A note on precision-preserving compression of scientific data R. Kouznetsov 10.5194/gmd-14-377-2021
19 citations as recorded by crossref.
- Advancing data compression via noise detection D. Hammerling & A. Baker 10.1038/s43588-021-00167-z
- Discussion on “Saving Storage in Climate Ensembles: A Model-Based Stochastic Approach” J. Bessac et al. 10.1007/s13253-023-00540-7
- Black-box statistical prediction of lossy compression ratios for scientific data R. Underwood et al. 10.1177/10943420231179417
- The impact of altering emission data precision on compression efficiency and accuracy of simulations of the community multiscale air quality model M. Walters & D. Wong 10.5194/gmd-16-1179-2023
- Compressing atmospheric data into its real information content M. Klöwer et al. 10.1038/s43588-021-00156-2
- Array DBMS R. Zalipynis 10.14778/3476311.3476404
- Optimizing Error-Bounded Lossy Compression for Scientific Data With Diverse Constraints Y. Liu et al. 10.1109/TPDS.2022.3194695
- The compression–error trade-off for large gridded data sets J. Silver & C. Zender 10.5194/gmd-10-413-2017
- Improving Performance of SLAV Model for Medium Range Weather Prediction R. Fadeev et al. 10.1134/S1995080224603874
- Evaluation of lossless and lossy algorithms for the compression of scientific datasets in netCDF-4 or HDF5 files X. Delaunay et al. 10.5194/gmd-12-4099-2019
- Parallel Implementation of a PETSc-Based Framework for the General Curvilinear Coastal Ocean Model M. Valera et al. 10.3390/jmse7060185
- New Methods for Data Storage of Model Output from Ensemble Simulations P. Düben et al. 10.1175/MWR-D-18-0170.1
- Evaluating image quality measures to assess the impact of lossy data compression applied to climate simulation data A. Baker et al. 10.1111/cgf.13707
- Data-Driven Artificial Intelligence for Calibration of Hyperspectral Big Data V. Sagan et al. 10.1109/TGRS.2021.3091409
- A statistical analysis of lossily compressed climate model data A. Poppick et al. 10.1016/j.cageo.2020.104599
- GenomicScores: seamless access to genomewide position-specific scores from R and Bioconductor P. Puigdevall et al. 10.1093/bioinformatics/bty311
- Evaluating lossy data compression on climate simulation data within a large ensemble A. Baker et al. 10.5194/gmd-9-4381-2016
- Multifacets of lossy compression for scientific data in the Joint-Laboratory of Extreme Scale Computing F. Cappello et al. 10.1016/j.future.2024.05.022
- A note on precision-preserving compression of scientific data R. Kouznetsov 10.5194/gmd-14-377-2021
Saved (final revised paper)
Latest update: 20 Nov 2024
Short summary
We introduce Bit Grooming, a lossy compression algorithm that removes the bloat due to false precision, those bits and bytes beyond the meaningful precision of the data. Bit Grooming is statistically unbiased, applies to all floating-point numbers, and is easy to use. Bit Grooming reduces data storage requirements by 25–80 %. Unlike its best-known competitor Linear Packing, Bit Grooming imposes no software overhead on users, and guarantees its precision throughout the whole floating-point range.
We introduce Bit Grooming, a lossy compression algorithm that removes the bloat due to false...
