Articles | Volume 17, issue 24
https://doi.org/10.5194/gmd-17-8909-2024
© Author(s) 2024. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/gmd-17-8909-2024
© Author(s) 2024. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Development and technical paper
| 
19 Dec 2024
Development and technical paper |
| 19 Dec 2024
| 19 Dec 2024
The effect of lossy compression of numerical weather prediction data on data analysis: a case study using enstools-compression 2023.11
Meteorological Institute, Ludwig Maximilian University of Munich, 80333 Munich, Germany
Robert Redl
Meteorological Institute, Ludwig Maximilian University of Munich, 80333 Munich, Germany
Marc Rautenhaus
Hub of Computing and Data Science, Visual Data Analysis Group, Universität Hamburg, 20146 Hamburg, Germany
Tobias Selz
Meteorological Institute, Ludwig Maximilian University of Munich, 80333 Munich, Germany
Takumi Matsunobu
Meteorological Institute, Ludwig Maximilian University of Munich, 80333 Munich, Germany
Kameswar Rao Modali
Hub of Computing and Data Science, Visual Data Analysis Group, Universität Hamburg, 20146 Hamburg, Germany
George Craig
Meteorological Institute, Ludwig Maximilian University of Munich, 80333 Munich, Germany
Related authors
Francisco J. Doblas-Reyes, Jenni Kontkanen, Irina Sandu, Mario Acosta, Mohammed Hussam Al Turjmam, Ivan Alsina-Ferrer, Miguel Andrés-Martínez, Leo Arriola, Marvin Axness, Marc Batlle Martín, Peter Bauer, Tobias Becker, Daniel Beltrán, Sebastian Beyer, Hendryk Bockelmann, Pierre-Antoine Bretonnière, Sebastien Cabaniols, Silvia Caprioli, Miguel Castrillo, Aparna Chandrasekar, Suvarchal Cheedela, Victor Correal, Emanuele Danovaro, Paolo Davini, Jussi Enkovaara, Claudia Frauen, Barbara Früh, Aina Gaya Àvila, Paolo Ghinassi, Rohit Ghosh, Supriyo Ghosh, Iker González, Katherine Grayson, Matthew Griffith, Ioan Hadade, Christopher Haine, Carl Hartick, Utz-Uwe Haus, Shane Hearne, Heikki Järvinen, Bernat Jiménez, Amal John, Marlin Juchem, Thomas Jung, Jessica Kegel, Matthias Kelbling, Kai Keller, Bruno Kinoshita, Theresa Kiszler, Daniel Klocke, Lukas Kluft, Nikolay Koldunov, Tobias Kölling, Joonas Kolstela, Luis Kornblueh, Sergey Kosukhin, Aleksander Lacima-Nadolnik, Jeisson Javier Leal Rojas, Jonni Lehtiranta, Tuomas Lunttila, Anna Luoma, Pekka Manninen, Alexey Medvedev, Sebastian Milinski, Ali Omar Abdelazim Mohammed, Sebastian Müller, Devaraju Naryanappa, Natalia Nazarova, Sami Niemelä, Bimochan Niraula, Henrik Nortamo, Aleksi Nummelin, Matteo Nurisso, Pablo Ortega, Stella Paronuzzi, Xabier Pedruzo Bagazgoitia, Charles Pelletier, Carlos Peña, Suraj Polade, Himansu Pradhan, Rommel Quintanilla, Tiago Quintino, Thomas Rackow, Jouni Räisänen, Maqsood Mubarak Rajput, René Redler, Balthasar Reuter, Nuno Rocha Monteiro, Francesc Roura-Adserias, Silva Ruppert, Susan Sayed, Reiner Schnur, Tanvi Sharma, Dmitry Sidorenko, Outi Sievi-Korte, Albert Soret, Christian Steger, Bjorn Stevens, Jan Streffing, Jaleena Sunny, Luiggi Tenorio, Stephan Thober, Ulf Tigerstedt, Oriol Tinto, Juha Tonttila, Heikki Tuomenvirta, Lauri Tuppi, Ginka Van Thielen, Emanuele Vitali, Jost von Hardenberg, Ingo Wagner, Nils Wedi, Jan Wehner, Sven Willner, Xavier Yepes-Arbós, Florian Ziemen, and Janos Zimmermann
EGUsphere, https://doi.org/10.5194/egusphere-2025-2198, https://doi.org/10.5194/egusphere-2025-2198, 2025
This preprint is open for discussion and under review for Geoscientific Model Development (GMD).
Short summary
Short summary
The Climate Change Adaptation Digital Twin (Climate DT) pioneers the operationalisation of climate projections. The system produces global simulations with local granularity for adaptation decision-making. Applications are embedded to generate tailored indicators. A unified workflow orchestrates all components in several supercomputers. Data management ensures consistency and streaming enables real-time use. It is a complementary innovation to initiatives like CMIP, CORDEX, and climate services.
Ralf Döscher, Mario Acosta, Andrea Alessandri, Peter Anthoni, Thomas Arsouze, Tommi Bergman, Raffaele Bernardello, Souhail Boussetta, Louis-Philippe Caron, Glenn Carver, Miguel Castrillo, Franco Catalano, Ivana Cvijanovic, Paolo Davini, Evelien Dekker, Francisco J. Doblas-Reyes, David Docquier, Pablo Echevarria, Uwe Fladrich, Ramon Fuentes-Franco, Matthias Gröger, Jost v. Hardenberg, Jenny Hieronymus, M. Pasha Karami, Jukka-Pekka Keskinen, Torben Koenigk, Risto Makkonen, François Massonnet, Martin Ménégoz, Paul A. Miller, Eduardo Moreno-Chamarro, Lars Nieradzik, Twan van Noije, Paul Nolan, Declan O'Donnell, Pirkka Ollinaho, Gijs van den Oord, Pablo Ortega, Oriol Tintó Prims, Arthur Ramos, Thomas Reerink, Clement Rousset, Yohan Ruprich-Robert, Philippe Le Sager, Torben Schmith, Roland Schrödner, Federico Serva, Valentina Sicardi, Marianne Sloth Madsen, Benjamin Smith, Tian Tian, Etienne Tourigny, Petteri Uotila, Martin Vancoppenolle, Shiyu Wang, David Wårlind, Ulrika Willén, Klaus Wyser, Shuting Yang, Xavier Yepes-Arbós, and Qiong Zhang
Geosci. Model Dev., 15, 2973–3020, https://doi.org/10.5194/gmd-15-2973-2022, https://doi.org/10.5194/gmd-15-2973-2022, 2022
Short summary
Short summary
The Earth system model EC-Earth3 is documented here. Key performance metrics show physical behavior and biases well within the frame known from recent models. With improved physical and dynamic features, new ESM components, community tools, and largely improved physical performance compared to the CMIP5 version, EC-Earth3 represents a clear step forward for the only European community ESM. We demonstrate here that EC-Earth3 is suited for a range of tasks in CMIP6 and beyond.
Francisco J. Doblas-Reyes, Jenni Kontkanen, Irina Sandu, Mario Acosta, Mohammed Hussam Al Turjmam, Ivan Alsina-Ferrer, Miguel Andrés-Martínez, Leo Arriola, Marvin Axness, Marc Batlle Martín, Peter Bauer, Tobias Becker, Daniel Beltrán, Sebastian Beyer, Hendryk Bockelmann, Pierre-Antoine Bretonnière, Sebastien Cabaniols, Silvia Caprioli, Miguel Castrillo, Aparna Chandrasekar, Suvarchal Cheedela, Victor Correal, Emanuele Danovaro, Paolo Davini, Jussi Enkovaara, Claudia Frauen, Barbara Früh, Aina Gaya Àvila, Paolo Ghinassi, Rohit Ghosh, Supriyo Ghosh, Iker González, Katherine Grayson, Matthew Griffith, Ioan Hadade, Christopher Haine, Carl Hartick, Utz-Uwe Haus, Shane Hearne, Heikki Järvinen, Bernat Jiménez, Amal John, Marlin Juchem, Thomas Jung, Jessica Kegel, Matthias Kelbling, Kai Keller, Bruno Kinoshita, Theresa Kiszler, Daniel Klocke, Lukas Kluft, Nikolay Koldunov, Tobias Kölling, Joonas Kolstela, Luis Kornblueh, Sergey Kosukhin, Aleksander Lacima-Nadolnik, Jeisson Javier Leal Rojas, Jonni Lehtiranta, Tuomas Lunttila, Anna Luoma, Pekka Manninen, Alexey Medvedev, Sebastian Milinski, Ali Omar Abdelazim Mohammed, Sebastian Müller, Devaraju Naryanappa, Natalia Nazarova, Sami Niemelä, Bimochan Niraula, Henrik Nortamo, Aleksi Nummelin, Matteo Nurisso, Pablo Ortega, Stella Paronuzzi, Xabier Pedruzo Bagazgoitia, Charles Pelletier, Carlos Peña, Suraj Polade, Himansu Pradhan, Rommel Quintanilla, Tiago Quintino, Thomas Rackow, Jouni Räisänen, Maqsood Mubarak Rajput, René Redler, Balthasar Reuter, Nuno Rocha Monteiro, Francesc Roura-Adserias, Silva Ruppert, Susan Sayed, Reiner Schnur, Tanvi Sharma, Dmitry Sidorenko, Outi Sievi-Korte, Albert Soret, Christian Steger, Bjorn Stevens, Jan Streffing, Jaleena Sunny, Luiggi Tenorio, Stephan Thober, Ulf Tigerstedt, Oriol Tinto, Juha Tonttila, Heikki Tuomenvirta, Lauri Tuppi, Ginka Van Thielen, Emanuele Vitali, Jost von Hardenberg, Ingo Wagner, Nils Wedi, Jan Wehner, Sven Willner, Xavier Yepes-Arbós, Florian Ziemen, and Janos Zimmermann
EGUsphere, https://doi.org/10.5194/egusphere-2025-2198, https://doi.org/10.5194/egusphere-2025-2198, 2025
This preprint is open for discussion and under review for Geoscientific Model Development (GMD).
Short summary
Short summary
The Climate Change Adaptation Digital Twin (Climate DT) pioneers the operationalisation of climate projections. The system produces global simulations with local granularity for adaptation decision-making. Applications are embedded to generate tailored indicators. A unified workflow orchestrates all components in several supercomputers. Data management ensures consistency and streaming enables real-time use. It is a complementary innovation to initiatives like CMIP, CORDEX, and climate services.
Tim Radke, Susanne Fuchs, Christian Wilms, Iuliia Polkova, and Marc Rautenhaus
Geosci. Model Dev., 18, 1017–1039, https://doi.org/10.5194/gmd-18-1017-2025, https://doi.org/10.5194/gmd-18-1017-2025, 2025
Short summary
Short summary
In our study, we built upon previous work to investigate the patterns artificial intelligence (AI) learns to detect atmospheric features like tropical cyclones (TCs) and atmospheric rivers (ARs). As primary objective, we adopt a method to explain the AI used and investigate the plausibility of learned patterns. We find that plausible patterns are learned for both TCs and ARs. Hence, the chosen method is very useful for gaining confidence in the AI-based detection of atmospheric features.
Christoph Fischer, Andreas H. Fink, Elmar Schömer, Marc Rautenhaus, and Michael Riemer
Geosci. Model Dev., 17, 4213–4228, https://doi.org/10.5194/gmd-17-4213-2024, https://doi.org/10.5194/gmd-17-4213-2024, 2024
Short summary
Short summary
This study presents a method for identifying and tracking 3-D potential vorticity structures within African easterly waves (AEWs). Each identified structure is characterized by descriptors, including its 3-D position and orientation, which have been validated through composite comparisons. A trough-centric perspective on the descriptors reveals the evolution and distinct characteristics of AEWs. These descriptors serve as valuable statistical inputs for the study of AEW-related phenomena.
Konstantin Krüger, Andreas Schäfler, Martin Weissmann, and George C. Craig
Weather Clim. Dynam., 5, 491–509, https://doi.org/10.5194/wcd-5-491-2024, https://doi.org/10.5194/wcd-5-491-2024, 2024
Short summary
Short summary
Initial conditions of current numerical weather prediction models insufficiently represent the sharp vertical gradients across the midlatitude tropopause. Observation-space data assimilation output is used to study the influence of assimilated radiosondes on the tropopause. The radiosondes reduce systematic biases of the model background and sharpen temperature and wind gradients in the analysis. Tropopause sharpness is still underestimated in the analysis, which may impact weather forecasts.
Hyunju Jung, Peter Knippertz, Yvonne Ruckstuhl, Robert Redl, Tijana Janjic, and Corinna Hoose
Weather Clim. Dynam., 4, 1111–1134, https://doi.org/10.5194/wcd-4-1111-2023, https://doi.org/10.5194/wcd-4-1111-2023, 2023
Short summary
Short summary
A narrow rainfall belt in the tropics is an important feature for large-scale circulations and the global water cycle. The accurate simulation of this rainfall feature has been a long-standing problem, with the reasons behind that unclear. We present a novel diagnostic tool that allows us to disentangle processes important for rainfall, which changes due to modifications in model. Using our diagnostic tool, one can potentially identify sources of uncertainty in weather and climate models.
Christoph Neuhauser, Maicon Hieronymus, Michael Kern, Marc Rautenhaus, Annika Oertel, and Rüdiger Westermann
Geosci. Model Dev., 16, 4617–4638, https://doi.org/10.5194/gmd-16-4617-2023, https://doi.org/10.5194/gmd-16-4617-2023, 2023
Short summary
Short summary
Numerical weather prediction models rely on parameterizations for sub-grid-scale processes, which are a source of uncertainty. We present novel visual analytics solutions to analyze interactively the sensitivities of a selected prognostic variable to multiple model parameters along trajectories regarding similarities in temporal development and spatiotemporal relationships. The proposed workflow is applied to cloud microphysical sensitivities along coherent strongly ascending trajectories.
Andreas A. Beckert, Lea Eisenstein, Annika Oertel, Tim Hewson, George C. Craig, and Marc Rautenhaus
Geosci. Model Dev., 16, 4427–4450, https://doi.org/10.5194/gmd-16-4427-2023, https://doi.org/10.5194/gmd-16-4427-2023, 2023
Short summary
Short summary
We investigate the benefit of objective 3-D front detection with modern interactive visual analysis techniques for case studies of extra-tropical cyclones and comparisons of frontal structures between different numerical weather prediction models. The 3-D frontal structures show agreement with 2-D fronts from surface analysis charts and augment them in the vertical dimension. We see great potential for more complex studies of atmospheric dynamics and for operational weather forecasting.
Konstantin Krüger, Andreas Schäfler, Martin Wirth, Martin Weissmann, and George C. Craig
Atmos. Chem. Phys., 22, 15559–15577, https://doi.org/10.5194/acp-22-15559-2022, https://doi.org/10.5194/acp-22-15559-2022, 2022
Short summary
Short summary
A comprehensive data set of airborne lidar water vapour profiles is compared with ERA5 reanalyses for a robust characterization of the vertical structure of the mid-latitude lower-stratospheric moist bias. We confirm a moist bias of up to 55 % at 1.3 km altitude above the tropopause and uncover a decreasing bias beyond. Collocated O3 and H2O observations reveal a particularly strong bias in the mixing layer, indicating insufficiently modelled transport processes fostering the bias.
Takumi Matsunobu, Christian Keil, and Christian Barthlott
Weather Clim. Dynam., 3, 1273–1289, https://doi.org/10.5194/wcd-3-1273-2022, https://doi.org/10.5194/wcd-3-1273-2022, 2022
Short summary
Short summary
This study quantifies the impact of poorly constrained parameters used to represent aerosol–cloud–precipitation interactions on precipitation and cloud forecasts associated with uncertainties in input atmospheric states. Uncertainties in these parameters have a non-negligible impact on daily precipitation amount and largely change the amount of cloud. The comparison between different weather situations reveals that the impact becomes more important when convection is triggered by local effects.
Andreas Alexander Beckert, Lea Eisenstein, Annika Oertel, Timothy Hewson, George C. Craig, and Marc Rautenhaus
Weather Clim. Dynam. Discuss., https://doi.org/10.5194/wcd-2022-36, https://doi.org/10.5194/wcd-2022-36, 2022
Preprint withdrawn
Short summary
Short summary
This study revises and extends a previously presented 3-D objective front detection method and demonstrates its benefits to analyse weather dynamics in numerical simulation data. Based on two case studies of extratropical cyclones, we demonstrate the evaluation of conceptual models from dynamic meteorology, illustrate the benefits of our interactive analysis approach by comparing fronts in data with different model resolutions, and study the impact of convection on fronts.
Christoph Fischer, Andreas H. Fink, Elmar Schömer, Roderick van der Linden, Michael Maier-Gerber, Marc Rautenhaus, and Michael Riemer
Geosci. Model Dev., 15, 4447–4468, https://doi.org/10.5194/gmd-15-4447-2022, https://doi.org/10.5194/gmd-15-4447-2022, 2022
Short summary
Short summary
Potential vorticity (PV) analysis plays a central role in studying atmospheric dynamics. For example, anomalies in the PV field near the tropopause are linked to extreme weather events. In this study, an objective strategy to identify these anomalies is presented and evaluated. As a novel concept, it can be applied to three-dimensional (3-D) data sets. Supported by 3-D visualizations, we illustrate advantages of this new analysis over existing studies along a case study.
Raphael Kriegmair, Yvonne Ruckstuhl, Stephan Rasp, and George Craig
Nonlin. Processes Geophys., 29, 171–181, https://doi.org/10.5194/npg-29-171-2022, https://doi.org/10.5194/npg-29-171-2022, 2022
Short summary
Short summary
Our regional numerical weather prediction models run at kilometer-scale resolutions. Processes that occur at smaller scales not yet resolved contribute significantly to the atmospheric flow. We use a neural network (NN) to represent the unresolved part of physical process such as cumulus clouds. We test this approach on a simplified, yet representative, 1D model and find that the NN corrections vastly improve the model forecast up to a couple of days.
Ralf Döscher, Mario Acosta, Andrea Alessandri, Peter Anthoni, Thomas Arsouze, Tommi Bergman, Raffaele Bernardello, Souhail Boussetta, Louis-Philippe Caron, Glenn Carver, Miguel Castrillo, Franco Catalano, Ivana Cvijanovic, Paolo Davini, Evelien Dekker, Francisco J. Doblas-Reyes, David Docquier, Pablo Echevarria, Uwe Fladrich, Ramon Fuentes-Franco, Matthias Gröger, Jost v. Hardenberg, Jenny Hieronymus, M. Pasha Karami, Jukka-Pekka Keskinen, Torben Koenigk, Risto Makkonen, François Massonnet, Martin Ménégoz, Paul A. Miller, Eduardo Moreno-Chamarro, Lars Nieradzik, Twan van Noije, Paul Nolan, Declan O'Donnell, Pirkka Ollinaho, Gijs van den Oord, Pablo Ortega, Oriol Tintó Prims, Arthur Ramos, Thomas Reerink, Clement Rousset, Yohan Ruprich-Robert, Philippe Le Sager, Torben Schmith, Roland Schrödner, Federico Serva, Valentina Sicardi, Marianne Sloth Madsen, Benjamin Smith, Tian Tian, Etienne Tourigny, Petteri Uotila, Martin Vancoppenolle, Shiyu Wang, David Wårlind, Ulrika Willén, Klaus Wyser, Shuting Yang, Xavier Yepes-Arbós, and Qiong Zhang
Geosci. Model Dev., 15, 2973–3020, https://doi.org/10.5194/gmd-15-2973-2022, https://doi.org/10.5194/gmd-15-2973-2022, 2022
Short summary
Short summary
The Earth system model EC-Earth3 is documented here. Key performance metrics show physical behavior and biases well within the frame known from recent models. With improved physical and dynamic features, new ESM components, community tools, and largely improved physical performance compared to the CMIP5 version, EC-Earth3 represents a clear step forward for the only European community ESM. We demonstrate here that EC-Earth3 is suited for a range of tasks in CMIP6 and beyond.
Marcel Meyer, Iuliia Polkova, Kameswar Rao Modali, Laura Schaffer, Johanna Baehr, Stephan Olbrich, and Marc Rautenhaus
Weather Clim. Dynam., 2, 867–891, https://doi.org/10.5194/wcd-2-867-2021, https://doi.org/10.5194/wcd-2-867-2021, 2021
Short summary
Short summary
Novel techniques from computer science are used to study extreme weather events. Inspired by the interactive 3-D visual analysis of the recently released ERA5 reanalysis data, we improve commonly used metrics for measuring polar winter storms and outbreaks of cold air. The software (Met.3D) that we have extended and applied as part of this study is freely available and can be used generically for 3-D visualization of a broad variety of atmospheric processes in weather and climate data.
Yong Wang, Guang J. Zhang, Shaocheng Xie, Wuyin Lin, George C. Craig, Qi Tang, and Hsi-Yen Ma
Geosci. Model Dev., 14, 1575–1593, https://doi.org/10.5194/gmd-14-1575-2021, https://doi.org/10.5194/gmd-14-1575-2021, 2021
Short summary
Short summary
A stochastic deep convection parameterization is implemented into the US Department of Energy Energy Exascale Earth System Model Atmosphere Model version 1 (EAMv1). Compared to the default model, the well-known problem of
too much light rain and too little heavy rainis largely alleviated over the tropics with the stochastic scheme. Results from this study provide important insights into the model performance of EAMv1 when stochasticity is included in the deep convective parameterization.
Xavier Fettweis, Stefan Hofer, Uta Krebs-Kanzow, Charles Amory, Teruo Aoki, Constantijn J. Berends, Andreas Born, Jason E. Box, Alison Delhasse, Koji Fujita, Paul Gierz, Heiko Goelzer, Edward Hanna, Akihiro Hashimoto, Philippe Huybrechts, Marie-Luise Kapsch, Michalea D. King, Christoph Kittel, Charlotte Lang, Peter L. Langen, Jan T. M. Lenaerts, Glen E. Liston, Gerrit Lohmann, Sebastian H. Mernild, Uwe Mikolajewicz, Kameswarrao Modali, Ruth H. Mottram, Masashi Niwano, Brice Noël, Jonathan C. Ryan, Amy Smith, Jan Streffing, Marco Tedesco, Willem Jan van de Berg, Michiel van den Broeke, Roderik S. W. van de Wal, Leo van Kampenhout, David Wilton, Bert Wouters, Florian Ziemen, and Tobias Zolles
The Cryosphere, 14, 3935–3958, https://doi.org/10.5194/tc-14-3935-2020, https://doi.org/10.5194/tc-14-3935-2020, 2020
Short summary
Short summary
We evaluated simulated Greenland Ice Sheet surface mass balance from 5 kinds of models. While the most complex (but expensive to compute) models remain the best, the faster/simpler models also compare reliably with observations and have biases of the same order as the regional models. Discrepancies in the trend over 2000–2012, however, suggest that large uncertainties remain in the modelled future SMB changes as they are highly impacted by the meltwater runoff biases over the current climate.
Cited articles
Alted, F.: Why modern CPUs are starving and what can be done about it, CiSE, 12, 68–71, https://doi.org/10.1109/MCSE.2010.51, 2010. a
Ayachit, U., Geveci, B., Moreland, K., Patchett, J., and Ahrens, J.: The ParaView Visualization Application, in: High Performance Visualization, edited by Bethel, E. W., Childs, H., and Hansen, C., chap. 18, CRC Press, 383–400, https://doi.org/10.1201/b12985-31, 2012. a
Bachmann, K., Keil, C., and Weissmann, M.: Impact of radar data assimilation and orography on predictability of deep convection, Q. J. Roy. Meteor. Soc., 145, 117–130, https://doi.org/10.1002/qj.3412, 2018. a
Bachmann, K., Keil, C., Craig, G. C., Weissmann, M., and Welzbacher, C. A.: Predictability of Deep Convection in Idealized and Operational Forecasts: Effects of Radar Data Assimilation, Orography, and Synoptic Weather Regime, Mon. Weather Rev., 148, 63–81, https://doi.org/10.1175/mwr-d-19-0045.1, 2019. a
Baker, A. H., Xu, H., Dennis, J. M., Levy, M. N., Nychka, D., Mickelson, S. A., Edwards, J., Vertenstein, M., and Wegener, A.: A methodology for evaluating the impact of data compression on climate simulation data, in: Proceedings of the 23rd international symposium on High-performance parallel and distributed computing, HPDC'14: The 23rd International Symposium on High-Performance Parallel and Distributed Computing, 23–27 June 2014, Vancouver BC Canada, ACM Press, https://doi.org/10.1145/2600212.2600217, 2014. a, b, c
Baker, A. H., Hammerling, D. M., Mickelson, S. A., Xu, H., Stolpe, M. B., Naveau, P., Sanderson, B., Ebert-Uphoff, I., Samarasinghe, S., De Simone, F., Carbone, F., Gencarelli, C. N., Dennis, J. M., Kay, J. E., and Lindstrom, P.: Evaluating lossy data compression on climate simulation data within a large ensemble, Geosci. Model Dev., 9, 4381–4403, https://doi.org/10.5194/gmd-9-4381-2016, 2016. a, b, c
Baker, A. H., Xu, H., Hammerling, D. M., Li, S., and Clyne, J. P.: Toward a Multi-method Approach: Lossy Data Compression for Climate Simulation Data, in: Lecture Notes in Computer Science, Springer International Publishing, 30–42, https://doi.org/10.1007/978-3-319-67630-2_3, 2017. a, b
Baker, A. H., Hammerling, D. M., and Turton, T. L.: Evaluating image quality measures to assess the impact of lossy data compression applied to climate simulation data, Comput. Graph. Forum, 38, 517–528, https://doi.org/10.1111/cgf.13707, 2019. a
Baker, A. H., Pinard, A., and Hammerling, D. M.: On a Structural Similarity Index Approach for Floating-Point Data, IEEE T. Vis. Comput. Gr., 30, 6261–6274, https://doi.org/10.1109/TVCG.2023.3332843, 2024. a
Ballester-Ripoll, R. and Pajarola, R.: Lossy volume compression using Tucker truncation and thresholding, Vis. Comput., 32, 1433–1446, https://doi.org/10.1007/s00371-015-1130-y, 2015. a, b
Ballester-Ripoll, R., Lindstrom, P., and Pajarola, R.: TTHRESH: Tensor Compression for Multidimensional Visual Data, IEEE T. Vis. Comput. Gr., 26, 2891–2903, https://doi.org/10.1109/tvcg.2019.2904063, 2020. a, b
Balsa Rodríguez, M., Gobbetti, E., Iglesias Guitián, J., Makhinya, M., Marton, F., Pajarola, R., and Suter, S.: State-of-the-Art in Compressed GPU-Based Direct Volume Rendering: State-of-the-Art in Compressed GPU-Based DVR, Comput. Graph. Forum, 33, 77–100, https://doi.org/10.1111/cgf.12280, 2014. a
Ben-Kiki, O., Evans, C., and Ingerson, B.: YAML Ain't Markup Language (YAML™) Version 1.2, https://yaml.org/spec/1.2/spec.html (last access: 22 November 2023), 2009. a
Beyer, J., Hadwiger, M., and Pfister, H.: State-of-the-Art in GPU-Based Large-Scale Volume Visualization: GPU-Based Large-Scale Volume Visualization, Comput. Graph. Forum, 34, 13–37, https://doi.org/10.1111/cgf.12605, 2015. a
Blosc Development Team: Blosc: A blocking, shuffling and loss-less compression library, https://www.blosc.org, last access: 19 August 2022. a
Boutell, T.: PNG (Portable Network Graphics) Specification Version 1.0, Tech. Rep., https://doi.org/10.17487/rfc2083, 1997. a
Bray, T.: The JavaScript Object Notation (JSON) Data Interchange Format, RFC 8259, Internet Engineering Task Force, https://doi.org/10.17487/RFC8259, 2017. a
Burden, A., Burden, R., and Faires, J. D.: Numerical Analysis, CENGAGE Learning Custom Publishing, 10 edn., ISBN 9780357705315, 2015. a
Caron, J.: Compression by Bit-Shaving, https://web.archive.org/web/20141024232623/https://www.unidata.ucar.edu/blogs/developer/entry/compression_by_bit_shaving (last access: 22 November 2023), 2014. a
Collet, Y.: LZ4 Frame Format Description, GitHub Repository [code], https://github.com/lz4/lz4 (last access: 14 October 2024), 2020. a
Collet, Y. and Kucherawy, M.: Zstandard Compression and the 'application/zstd' Media Type, RFC 8878, https://doi.org/10.17487/RFC8878, 2021. a
Collette, A.: Python and HDF5, O'Reilly Media, Inc., ISBN 9781449367831, 2013. a
Craig, G. C., Fink, A. H., Hoose, C., Janjić, T., Knippertz, P., Laurian, A., Lerch, S., Mayer, B., Miltenberger, A., Redl, R., Riemer, M., Tempest, K. I., and Wirth, V.: Waves to Weather: Exploring the Limits of Predictability of Weather, B. Am. Meteorol. Soc., 102, E2151–E2164, https://doi.org/10.1175/BAMS-D-20-0035.1, 2021. a, b
Delaunay, X., Courtois, A., and Gouillon, F.: Evaluation of lossless and lossy algorithms for the compression of scientific datasets in netCDF-4 or HDF5 files, Geosci. Model Dev., 12, 4099–4113, https://doi.org/10.5194/gmd-12-4099-2019, 2019. a, b
Deutsch, P. and Gailly, J.: RFC 1950: ZLIB Compressed Data Format Specification, Internet Engineering Task Force (IETF), https://doi.org/10.17487/RFC1950, 1996. a
Dey, S. R. A., Leoncini, G., Roberts, N. M., Plant, R. S., and Migliorini, S.: A Spatial View of Ensemble Spread in Convection Permitting Ensembles, Mon. Weather Rev., 142, 4091–4107, https://doi.org/10.1175/mwr-d-14-00172.1, 2014. a
Donayre Holtz, S.: Lossy Compression of Climate Data Using Convolutional Autoencoders, Master Thesis, Karlsruher Institut für Technologie (KIT), https://doi.org/10.5445/IR/1000144742, 2022. a
Düben, P. D., Leutbecher, M., and Bauer, P.: New Methods for Data Storage of Model Output from Ensemble Simulations, Mon. Weather Rev., 147, 677–689, https://doi.org/10.1175/mwr-d-18-0170.1, 2019. a, b
enstools-compression Contributors: enstools-compression Documentation, https://enstools-compression.readthedocs.io/en/latest/, last access: 22 November 2023. a
Gneiting, T., Westveld, A. H., and Goldman, T.: Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation, Mon. Weather Rev., 133, 1098–1118, https://doi.org/10.1175/MWR2904.1, 2005. a
Gong, Q., Chen, J., Whitney, B., Liang, X., Reshniak, V., Banerjee, T., Lee, J., Rangarajan, A., Wan, L., Vidal, N., Liu, Q., Gainaru, A., Podhorszki, N., Archibald, R., Ranka, S., and Klasky, S.: MGARD: A multigrid framework for high-performance, error-controlled data compression and refactoring, SoftwareX, 24, 101590, https://doi.org/10.1016/j.softx.2023.101590, 2023. a
h5netcdf Contributors: h5netcdf: A Python interface for the netCDF4 file-format based on h5py, https://h5netcdf.org, last access: 22 November 2023. a
Hersbach, H., Bell, B., Berrisford, P., Blavati, G., Horányi, A., Muñoz, J. S., Nicolas, J., Peubey, C., Radu, R., Rozum, I., Schepers, D., Simmons, A., Soci, C., Dee, D., and Thépaut, J.-N.: ERA5 hourly data on pressure levels from 1959 to present, Copernicus Climate Change Service (C3S) Climate Data Store (CDS) [data set], https://doi.org/10.24381/cds.bd0915c6, 2018. a, b
Hoang, D., Klacansky, P., Bhatia, H., Bremer, P.-T., Lindstrom, P., and Pascucci, V.: A Study of the Trade-off Between Reducing Precision and Reducing Resolution for Data Analysis and Visualization, IEEE T. Vis. Comput. Gr., 25, 1193–1203, https://doi.org/10.1109/TVCG.2018.2864853, 2019. a, b, c
Hoang, D., Summa, B., Bhatia, H., Lindstrom, P., Klacansky, P., Usher, W., Bremer, P.-T., and Pascucci, V.: Efficient and Flexible Hierarchical Data Layouts for a Unified Encoding of Scalar Field Precision and Resolution, IEEE T. Vis. Comput. Gr., 27, 603–613, https://doi.org/10.1109/TVCG.2020.3030381, 2021. a
Hoyer, S. and Hamman, J.: xarray: N-D labeled Arrays and Datasets in Python, Journal of Open Research Software, 5, 10, https://doi.org/10.5334/jors.148, 2017. a
Kern, M., Hewson, T., Sadlo, F., Westermann, R., and Rautenhaus, M.: Robust Detection and Visualization of Jet-Stream Core Lines in Atmospheric Flow, IEEE T. Vis. Comput. Gr., 24, 893–902, https://doi.org/10.1109/tvcg.2017.2743989, 2018. a, b
Kochenderfer, M. J. and Wheeler, T. A.: Algorithms for Optimization, The MIT Press, ISBN 9780262039420, 2019. a
Koranne, S.: Hierarchical Data Format 5: HDF5, in: Handbook of Open Source Tools, Springer US, 191–200, https://doi.org/10.1007/978-1-4419-7719-9_10, 2010. a
Kunkel, J., Novikova, A., Betke, E., and Schaare, A.: Toward Decoupling the Selection of Compression Algorithms from Quality Constraints, in: Lecture Notes in Computer Science, Springer International Publishing, 3–14, https://doi.org/10.1007/978-3-319-67630-2_1, 2017. a, b
Lawrence, B. N., Kunkel, J. M., Churchill, J., Massey, N., Kershaw, P., and Pritchard, M.: Beating data bottlenecks in weather and climate science, in: Extreme Data Workshop 2018 Forschungszentrum Jülich, 18–19 September 2018 Proceedings, edited by: Schultz, M., Pleiter, D., and Bauer, P., vol. 40 of Schriften des Forschungszentrums Jülich IAS Series, Forschungszentrum Jülich GmbH Zentralbibliothek, Verlag Jülich, 18–19, ISBN 978-3-95806-392-1, https://pdfs.semanticscholar.org/9881/ed9d9e16cb70fba9456fb0905bf28c450ce0.pdf#page=38 (last access: 14 October 2024), 2019. a
Li, S., Jaroszynski, S., Pearse, S., Orf, L., and Clyne, J.: VAPOR: A Visualization Package Tailored to Analyze Simulation Data in Earth System Science, Atmosphere, 10, 488, https://doi.org/10.3390/atmos10090488, 2019. a
Li, S., Lindstrom, P., and Clyne, J.: Lossy Scientific Data Compression With SPERR, in: 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS), 37th IEEE International Parallel & Distributed Processing Symposium, 15–19 May 2023, Hilton St. Petersburg Bayfront Hotel St. Petersburg, Florida USA, 1007–1017, https://doi.org/10.1109/IPDPS54959.2023.00104, 2023. a
Liang, X., Di, S., Tao, D., Li, S., Li, S., Guo, H., Chen, Z., and Cappello, F.: Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets, in: 2018 IEEE International Conference on Big Data (Big Data), 2–7 July 2018, San Francisco, CA, USA, https://doi.org/10.1109/bigdata.2018.8622520, 2018. a, b, c, d
Liang, X., Guo, H., Di, S., Cappello, F., Raj, M., Liu, C., Ono, K., Chen, Z., and Peterka, T.: Toward Feature-Preserving 2D and 3D Vector Field Compression, in: 2020 IEEE Pacific Visualization Symposium (PacificVis), 14–17 April 2020, Tianjin, China, 81–90, ISBN 978-1-72815-697-2, https://doi.org/10.1109/PacificVis48177.2020.6431, 2020. a
Lindstrom, P. and Isenburg, M.: Fast and Efficient Compression of Floating-Point Data, IEEE T. Vis. Comput. Gr., 12, 1245–1250, https://doi.org/10.1109/tvcg.2006.143, 2006. a
Liu, J., Di, S., Zhao, K., Jin, S., Tao, D., Liang, X., Chen, Z., and Cappello, F.: Exploring Autoencoder-based Error-bounded Compression for Scientific Data, 2021 IEEE International Conference on Cluster Computing (CLUSTER), 7–10 September 2021, Portland, OR, USA, https://doi.org/10.1109/Cluster48925.2021.00034, 2021. a, b, c, d
Lu, Y., Jiang, K., Levine, J. A., and Berger, M.: Compressive Neural Representations of Volumetric Scalar Fields, Comput. Graph. Forum, 40, 135–146, https://doi.org/10.1111/cgf.14295, 2021. a
Matsunobu, T., Keil, C., and Barthlott, C.: The impact of microphysical uncertainty conditional on initial and boundary condition uncertainty under varying synoptic control, Weather Clim. Dynam., 3, 1273–1289, https://doi.org/10.5194/wcd-3-1273-2022, 2022. a, b
Matsunobu, T., Puh, M., and Keil, C.: Flow- and scale-dependent spatial predictability of convective precipitation combining different model uncertainty representations, Q. J. Roy. Meteor. Soc., 150, 2364–2381, https://doi.org/10.1002/qj.4713, 2024. a
Norton, A. and Clyne, J.: The VAPOR Visualization Application, in: High Performance Visualization, edited by Bethel, E. W., Childs, H., and Hansen, C., chap. 20, CRC Press, 415–428, https://doi.org/10.1201/b12985-33, 2012. a
Poppick, A., Nardi, J., Feldman, N., Baker, A. H., Pinard, A., and Hammerling, D. M.: A statistical analysis of lossily compressed climate model data, Comput. Geosci., 145, 104599, https://doi.org/10.1016/j.cageo.2020.104599, 2020. a, b
Rautenhaus, M., Grams, C. M., Schäfler, A., and Westermann, R.: Three-dimensional visualization of ensemble weather forecasts – Part 2: Forecasting warm conveyor belt situations for aircraft-based field campaigns, Geosci. Model Dev., 8, 2355–2377, https://doi.org/10.5194/gmd-8-2355-2015, 2015a. a
Rautenhaus, M., Kern, M., Schäfler, A., and Westermann, R.: Three-dimensional visualization of ensemble weather forecasts – Part 1: The visualization tool Met.3D (version 1.0), Geosci. Model Dev., 8, 2329–2353, https://doi.org/10.5194/gmd-8-2329-2015, 2015b. a, b
Rautenhaus, M., Bottinger, M., Siemen, S., Hoffman, R., Kirby, R. M., Mirzargar, M., Rober, N., and Westermann, R.: Visualization in Meteorology – A Survey of Techniques and Tools for Data Analysis Tasks, IEEE T. Vis. Comput. Gr., 24, 3268–3296, https://doi.org/10.1109/tvcg.2017.2779501, 2018. a, b, c
Redl, R., Tintó-Prims, O., baurflorian, and cagau: wavestoweather/enstools: Release v2022.11.1, Zenodo [code], https://doi.org/10.5281/zenodo.7331674, 2022. a, b, c
Rew, R., Davis, G., Emmerson, S., Cormack, C., Caron, J., Pincus, R., Hartnett, E., Heimbigner, D., Appel, L., and Fisher, W.: Unidata NetCDF, NSF [software], https://doi.org/10.5065/D6H70CW6, 1989. a
Roberts, N. M. and Lean, H. W.: Scale-Selective Verification of Rainfall Accumulations from High-Resolution Forecasts of Convective Events, Mon. Weather Rev., 136, 78–97, https://doi.org/10.1175/2007mwr2123.1, 2008. a
Schäfler, A., Craig, G., Wernli, H., Arbogast, P., Doyle, J. D., McTaggart-Cowan, R., Methven, J., Rivière, G., Ament, F., Boettcher, M., Bramberger, M., Cazenave, Q., Cotton, R., Crewell, S., Delanoë, J., Dörnbrack, A., Ehrlich, A., Ewald, F., Fix, A., Grams, C. M., Gray, S. L., Grob, H., Groß, S., Hagen, M., Harvey, B., Hirsch, L., Jacob, M., Kölling, T., Konow, H., Lemmerz, C., Lux, O., Magnusson, L., Mayer, B., Mech, M., Moore, R., Pelon, J., Quinting, J., Rahm, S., Rapp, M., Rautenhaus, M., Reitebuch, O., Reynolds, C. A., Sodemann, H., Spengler, T., Vaughan, G., Wendisch, M., Wirth, M., Witschas, B., Wolf, K., and Zinner, T.: The North Atlantic Waveguide and Downstream Impact Experiment, B. Am. Meteorol. Soc., 99, 1607–1637, https://doi.org/10.1175/bams-d-17-0003.1, 2018. a
Selz, T., Riemer, M., and Craig, G. C.: The Transition from Practical to Intrinsic Predictability of Midlatitude Weather, J. Atmos. Sci., 79, 2013–2030, https://doi.org/10.1175/jas-d-21-0271.1, 2022. a, b, c, d
Sprenger, M. and Wernli, H.: The LAGRANTO Lagrangian analysis tool – version 2.0, Geosci. Model Dev., 8, 2569–2586, https://doi.org/10.5194/gmd-8-2569-2015, 2015. a
Tao, D., Di, S., Guo, H., Chen, Z., and Cappello, F.: Z-checker: A framework for assessing lossy compression of scientific data, Int. J. High Perform. Comput. Appl., 33, 285–303, https://doi.org/10.1177/1094342017737147, 2019a. a
Tao, D., Di, S., Liang, X., Chen, Z., and Cappello, F.: Optimizing Lossy Compression Rate-Distortion from Automatic Online Selection between SZ and ZFP, IEEE Trans. Parallel Distrib. Syst., 30, 1857–1871, https://doi.org/10.1109/TPDS.2019.2894404, 2019b. a, b
The HDF Group: HDF5 Filters, https://support.hdfgroup.org/documentation/hdf5/latest/group___h5_z.html (last access: 14 October 2024), 2023. a
Tintó Prims, O.: wavestoweather/enstools-compression: Release v2023.11.1, Zenodo [code], https://doi.org/10.5281/zenodo.7327682, 2023. a
Tintó Prims, O.: The effect of lossy compression of numerical weather prediction data on data analysis: software to reproduce figures using enstools-compression, Zenodo [software], https://doi.org/10.5281/zenodo.10998604, 2024. a
Underwood, R., Malvoso, V., Calhoun, J. C., Di, S., and Cappello, F.: Productive and Performant Generic Lossy Data Compression with LibPressio, in: 2021 7th International Workshop on Data Analysis and Reduction for Big Scientific Data (DRBSD-7), 14 November 2021, St. Louis, Missouri, USA, 1–10, https://doi.org/10.1109/DRBSD754563.2021.00005, 2021. a
Vincent, T., V. Armando Solé, Kieffer, J., Kittisopikul, M., Florian-G, Plaswig, F., Valls, V., Klein, J., Gerstel, M., Junyuewang, and Payno: silx-kit/hdf5plugin: 3.3.1: 2022/06/03, Zenodo [code], https://doi.org/10.5281/zenodo.7257761, 2022. a, b
Wang, Z., Bovik, A., Sheikh, H., and Simoncelli, E.: Image Quality Assessment: From Error Visibility to Structural Similarity, IEEE Trans. Image Process., 13, 600–612, https://doi.org/10.1109/tip.2003.819861, 2004. a, b
Weiss, S., Hermüller, P., and Westermann, R.: Fast Neural Representations for Direct Volume Rendering, Comput. Graph. Forum, 41, 196–211, https://doi.org/10.1111/cgf.14578, 2022. a
Zender, C. S.: Bit Grooming: statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+), Geosci. Model Dev., 9, 3199–3211, https://doi.org/10.5194/gmd-9-3199-2016, 2016. a, b, c, d
Zhao, K., Di, S., Lian, X., Li, S., Tao, D., Bessac, J., Chen, Z., and Cappello, F.: SDRBench: Scientific Data Reduction Benchmark for Lossy Compressors, in: 2020 IEEE International Conference on Big Data (Big Data), IEEE, 10–13 December 2020, online, https://doi.org/10.1109/bigdata50022.2020.9378449, 2020. a, b, c, d, e, f, g, h
Short summary
Advanced compression techniques can drastically reduce the size of meteorological datasets (by 5 to 150 times) without compromising the data's scientific value. We developed a user-friendly tool called
enstools-compressionthat makes this compression simple for Earth scientists. This tool works seamlessly with common weather and climate data formats. Our work shows that lossy compression can significantly improve how researchers store and analyze large meteorological datasets.
Advanced compression techniques can drastically reduce the size of meteorological datasets (by 5...
