Articles | Volume 10, issue 1
https://doi.org/10.5194/gmd-10-413-2017
https://doi.org/10.5194/gmd-10-413-2017
Development and technical paper
 | 
27 Jan 2017
Development and technical paper |  | 27 Jan 2017

The compression–error trade-off for large gridded data sets

Jeremy D. Silver and Charles S. Zender

Related authors

Interannual variability in the Australian carbon cycle over 2015–2019, based on assimilation of Orbiting Carbon Observatory-2 (OCO-2) satellite data
Yohanna Villalobos, Peter J. Rayner, Jeremy D. Silver, Steven Thomas, Vanessa Haverd, Jürgen Knauer, Zoë M. Loh, Nicholas M. Deutscher, David W. T. Griffith, and David F. Pollard
Atmos. Chem. Phys., 22, 8897–8934, https://doi.org/10.5194/acp-22-8897-2022,https://doi.org/10.5194/acp-22-8897-2022, 2022
Short summary
Was Australia a sink or source of CO2 in 2015? Data assimilation using OCO-2 satellite measurements
Yohanna Villalobos, Peter J. Rayner, Jeremy D. Silver, Steven Thomas, Vanessa Haverd, Jürgen Knauer, Zoë M. Loh, Nicholas M. Deutscher, David W. T. Griffith, and David F. Pollard
Atmos. Chem. Phys., 21, 17453–17494, https://doi.org/10.5194/acp-21-17453-2021,https://doi.org/10.5194/acp-21-17453-2021, 2021
Short summary
Comparison of formaldehyde tropospheric columns in Australia and New Zealand using MAX-DOAS, FTIR and TROPOMI
Robert G. Ryan, Jeremy D. Silver, Richard Querel, Dan Smale, Steve Rhodes, Matt Tully, Nicholas Jones, and Robyn Schofield
Atmos. Meas. Tech., 13, 6501–6519, https://doi.org/10.5194/amt-13-6501-2020,https://doi.org/10.5194/amt-13-6501-2020, 2020
Short summary
The potential of Orbiting Carbon Observatory-2 data to reduce the uncertainties in CO2 surface fluxes over Australia using a variational assimilation scheme
Yohanna Villalobos, Peter Rayner, Steven Thomas, and Jeremy Silver
Atmos. Chem. Phys., 20, 8473–8500, https://doi.org/10.5194/acp-20-8473-2020,https://doi.org/10.5194/acp-20-8473-2020, 2020
Short summary
Estimating global gross primary productivity using chlorophyll fluorescence and a data assimilation system with the BETHY-SCOPE model
Alexander J. Norton, Peter J. Rayner, Ernest N. Koffi, Marko Scholze, Jeremy D. Silver, and Ying-Ping Wang
Biogeosciences, 16, 3069–3093, https://doi.org/10.5194/bg-16-3069-2019,https://doi.org/10.5194/bg-16-3069-2019, 2019
Short summary

Related subject area

Numerical methods
Numerical stabilization methods for level-set-based ice front migration
Gong Cheng, Mathieu Morlighem, and G. Hilmar Gudmundsson
Geosci. Model Dev., 17, 6227–6247, https://doi.org/10.5194/gmd-17-6227-2024,https://doi.org/10.5194/gmd-17-6227-2024, 2024
Short summary
Modelling chemical advection during magma ascent
Hugo Dominguez, Nicolas Riel, and Pierre Lanari
Geosci. Model Dev., 17, 6105–6122, https://doi.org/10.5194/gmd-17-6105-2024,https://doi.org/10.5194/gmd-17-6105-2024, 2024
Short summary
Consistent point data assimilation in Firedrake and Icepack
Reuben W. Nixon-Hill, Daniel Shapero, Colin J. Cotter, and David A. Ham
Geosci. Model Dev., 17, 5369–5386, https://doi.org/10.5194/gmd-17-5369-2024,https://doi.org/10.5194/gmd-17-5369-2024, 2024
Short summary
A computationally efficient parameterization of aerosol, cloud and precipitation pH for application at global and regional scale (EQSAM4Clim-v12)
Swen Metzger, Samuel Rémy, Jason E. Williams, Vincent Huijnen, and Johannes Flemming
Geosci. Model Dev., 17, 5009–5021, https://doi.org/10.5194/gmd-17-5009-2024,https://doi.org/10.5194/gmd-17-5009-2024, 2024
Short summary
Assessing the benefits of approximately exact step sizes for Picard and Newton solver in simulating ice flow (FEniCS-full-Stokes v.1.3.2)
Niko Schmidt, Angelika Humbert, and Thomas Slawig
Geosci. Model Dev., 17, 4943–4959, https://doi.org/10.5194/gmd-17-4943-2024,https://doi.org/10.5194/gmd-17-4943-2024, 2024
Short summary

Cited articles

Baker, A. H., Xu, H., Dennis, J. M., Levy, M. N., Nychka, D., Mickelson, S. A., Edwards, J., Vertenstein, M., and Wegener, A.: A methodology for evaluating the impact of data compression on climate simulation data, in: Proceedings of the 23rd International Symposium on High-performance Parallel and Distributed Computing, Vancouver, BC, Canada, 23–27 June 2014, ACM, 203–214, 2014.
Berger, T.: Rate-Distortion Theory, in: Wiley Encyclopedia of Telecommunications, John Wiley & Sons, Inc., https://doi.org/10.1002/0471219282.eot142, 2003.
Caron, J.: Converting GRIB to netCDF-4: Compression studies, presentation to the workshop "Closing the GRIB/netCDF gap", European Centre for Medium Range Weather Forecasts (ECMWF), 24–25 September 2014, Reading, UK, available at: http://www.ecmwf.int/sites/default/files/elibrary/2014/13711-converting-grib-netcdf-4.pdf, last access: 17 June 2016, 2014.
Dee, D., Uppala, S., Simmons, A., Berrisford, P., Poli, P., Kobayashi, S., Andrae, U., Balmaseda, M., Balsamo, G., Bauer, P., and Bechtold, P.: The ERA-Interim reanalysis: Configuration and performance of the data assimilation system, Q. J. Roy. Meteor. Soc., 137, 553–597, 2011.
Deutsch, L. P.: DEFLATE compressed data format specification version 1.3, Tech. Rep. IETF RFC1951, Internet Engineering Task Force, 2008.
Download
Short summary
Many modern scientific research projects generate large amounts of data. Storage space is valuable and may be limited; hence compression is vital. We tested different compression methods for large gridded data sets, assessing the space savings and the amount of precision lost. We found a general trade-off between precision and compression, with compression well-predicted by the entropy of the data set. A method introduced here proved to be a competitive archive format for gridded numerical data.