Articles | Volume 10, issue 1
https://doi.org/10.5194/gmd-10-413-2017
https://doi.org/10.5194/gmd-10-413-2017
Development and technical paper
 | 
27 Jan 2017
Development and technical paper |  | 27 Jan 2017

The compression–error trade-off for large gridded data sets

Jeremy D. Silver and Charles S. Zender

Related authors

Interannual variability in the Australian carbon cycle over 2015–2019, based on assimilation of Orbiting Carbon Observatory-2 (OCO-2) satellite data
Yohanna Villalobos, Peter J. Rayner, Jeremy D. Silver, Steven Thomas, Vanessa Haverd, Jürgen Knauer, Zoë M. Loh, Nicholas M. Deutscher, David W. T. Griffith, and David F. Pollard
Atmos. Chem. Phys., 22, 8897–8934, https://doi.org/10.5194/acp-22-8897-2022,https://doi.org/10.5194/acp-22-8897-2022, 2022
Short summary
Was Australia a sink or source of CO2 in 2015? Data assimilation using OCO-2 satellite measurements
Yohanna Villalobos, Peter J. Rayner, Jeremy D. Silver, Steven Thomas, Vanessa Haverd, Jürgen Knauer, Zoë M. Loh, Nicholas M. Deutscher, David W. T. Griffith, and David F. Pollard
Atmos. Chem. Phys., 21, 17453–17494, https://doi.org/10.5194/acp-21-17453-2021,https://doi.org/10.5194/acp-21-17453-2021, 2021
Short summary
Comparison of formaldehyde tropospheric columns in Australia and New Zealand using MAX-DOAS, FTIR and TROPOMI
Robert G. Ryan, Jeremy D. Silver, Richard Querel, Dan Smale, Steve Rhodes, Matt Tully, Nicholas Jones, and Robyn Schofield
Atmos. Meas. Tech., 13, 6501–6519, https://doi.org/10.5194/amt-13-6501-2020,https://doi.org/10.5194/amt-13-6501-2020, 2020
Short summary
The potential of Orbiting Carbon Observatory-2 data to reduce the uncertainties in CO2 surface fluxes over Australia using a variational assimilation scheme
Yohanna Villalobos, Peter Rayner, Steven Thomas, and Jeremy Silver
Atmos. Chem. Phys., 20, 8473–8500, https://doi.org/10.5194/acp-20-8473-2020,https://doi.org/10.5194/acp-20-8473-2020, 2020
Short summary
Estimating global gross primary productivity using chlorophyll fluorescence and a data assimilation system with the BETHY-SCOPE model
Alexander J. Norton, Peter J. Rayner, Ernest N. Koffi, Marko Scholze, Jeremy D. Silver, and Ying-Ping Wang
Biogeosciences, 16, 3069–3093, https://doi.org/10.5194/bg-16-3069-2019,https://doi.org/10.5194/bg-16-3069-2019, 2019
Short summary

Related subject area

Numerical methods
ParticleDA.jl v.1.0: a distributed particle-filtering data assimilation package
Daniel Giles, Matthew M. Graham, Mosè Giordano, Tuomas Koskela, Alexandros Beskos, and Serge Guillas
Geosci. Model Dev., 17, 2427–2445, https://doi.org/10.5194/gmd-17-2427-2024,https://doi.org/10.5194/gmd-17-2427-2024, 2024
Short summary
HETerogeneous vectorized or Parallel (HETPv1.0): an updated inorganic heterogeneous chemistry solver for the metastable-state NH4+–Na+–Ca2+–K+–Mg2+–SO42−–NO3–Cl–H2O system based on ISORROPIA II
Stefan J. Miller, Paul A. Makar, and Colin J. Lee
Geosci. Model Dev., 17, 2197–2219, https://doi.org/10.5194/gmd-17-2197-2024,https://doi.org/10.5194/gmd-17-2197-2024, 2024
Short summary
Three-dimensional geological modelling of igneous intrusions in LoopStructural v1.5.10
Fernanda Alvarado-Neves, Laurent Ailleres, Lachlan Grose, Alexander R. Cruden, and Robin Armit
Geosci. Model Dev., 17, 1975–1993, https://doi.org/10.5194/gmd-17-1975-2024,https://doi.org/10.5194/gmd-17-1975-2024, 2024
Short summary
Estimating volcanic ash emissions using retrieved satellite ash columns and inverse ash transport modeling using VolcanicAshInversion v1.2.1, within the operational eEMEP (emergency European Monitoring and Evaluation Programme) volcanic plume forecasting system (version rv4_17)
André R. Brodtkorb, Anna Benedictow, Heiko Klein, Arve Kylling, Agnes Nyiri, Alvaro Valdebenito, Espen Sollum, and Nina Kristiansen
Geosci. Model Dev., 17, 1957–1974, https://doi.org/10.5194/gmd-17-1957-2024,https://doi.org/10.5194/gmd-17-1957-2024, 2024
Short summary
Accounting for uncertainties in forecasting tropical-cyclone-induced compound flooding
Kees Nederhoff, Maarten van Ormondt, Jay Veeramony, Ap van Dongeren, José Antonio Álvarez Antolínez, Tim Leijnse, and Dano Roelvink
Geosci. Model Dev., 17, 1789–1811, https://doi.org/10.5194/gmd-17-1789-2024,https://doi.org/10.5194/gmd-17-1789-2024, 2024
Short summary

Cited articles

Baker, A. H., Xu, H., Dennis, J. M., Levy, M. N., Nychka, D., Mickelson, S. A., Edwards, J., Vertenstein, M., and Wegener, A.: A methodology for evaluating the impact of data compression on climate simulation data, in: Proceedings of the 23rd International Symposium on High-performance Parallel and Distributed Computing, Vancouver, BC, Canada, 23–27 June 2014, ACM, 203–214, 2014.
Berger, T.: Rate-Distortion Theory, in: Wiley Encyclopedia of Telecommunications, John Wiley & Sons, Inc., https://doi.org/10.1002/0471219282.eot142, 2003.
Caron, J.: Converting GRIB to netCDF-4: Compression studies, presentation to the workshop "Closing the GRIB/netCDF gap", European Centre for Medium Range Weather Forecasts (ECMWF), 24–25 September 2014, Reading, UK, available at: http://www.ecmwf.int/sites/default/files/elibrary/2014/13711-converting-grib-netcdf-4.pdf, last access: 17 June 2016, 2014.
Dee, D., Uppala, S., Simmons, A., Berrisford, P., Poli, P., Kobayashi, S., Andrae, U., Balmaseda, M., Balsamo, G., Bauer, P., and Bechtold, P.: The ERA-Interim reanalysis: Configuration and performance of the data assimilation system, Q. J. Roy. Meteor. Soc., 137, 553–597, 2011.
Deutsch, L. P.: DEFLATE compressed data format specification version 1.3, Tech. Rep. IETF RFC1951, Internet Engineering Task Force, 2008.
Download
Short summary
Many modern scientific research projects generate large amounts of data. Storage space is valuable and may be limited; hence compression is vital. We tested different compression methods for large gridded data sets, assessing the space savings and the amount of precision lost. We found a general trade-off between precision and compression, with compression well-predicted by the entropy of the data set. A method introduced here proved to be a competitive archive format for gridded numerical data.