Articles | Volume 17, issue 23
https://doi.org/10.5194/gmd-17-8593-2024
© Author(s) 2024. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/gmd-17-8593-2024
© Author(s) 2024. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Development and technical paper
| 
05 Dec 2024
Development and technical paper |
| 05 Dec 2024
| 05 Dec 2024
Software sustainability of global impact models
Emmanuel Nyenah
CORRESPONDING AUTHOR
Institute of Physical Geography, Goethe University Frankfurt, 60438 Frankfurt am Main, Germany
Petra Döll
Institute of Physical Geography, Goethe University Frankfurt, 60438 Frankfurt am Main, Germany
Senckenberg Biodiversity and Climate Research Centre (SBiK-F), 60438 Frankfurt am Main, Germany
Daniel S. Katz
NCSA, CS, ECE and iSchool, University of Illinois Urbana-Champaign, Urbana, IL 61801, USA
Robert Reinecke
Institute of Geography, Johannes Gutenberg University Mainz, 55128 Mainz, Germany
Related authors
Emmanuel Nyenah, Petra Döll, Martina Flörke, Leon Mühlenbruch, Lasse Nissen, and Robert Reinecke
Geosci. Model Dev., 18, 5635–5653, https://doi.org/10.5194/gmd-18-5635-2025, https://doi.org/10.5194/gmd-18-5635-2025, 2025
Short summary
Short summary
We reprogrammed the latest WaterGAP model (2.2e) to create a sustainable global hydrological model. By utilizing best software practices like modular design, version control, and clear documentation, the new WaterGAP supports collaboration across teams. It can be easily understood, applied, and enhanced by both novice and experienced modellers. Additionally, we share the reprogramming process to assist in the reprogramming of other large geoscientific research software.
Emmanuel Nyenah, Petra Döll, Martina Flörke, Leon Mühlenbruch, Lasse Nissen, and Robert Reinecke
Geosci. Model Dev., 18, 5635–5653, https://doi.org/10.5194/gmd-18-5635-2025, https://doi.org/10.5194/gmd-18-5635-2025, 2025
Short summary
Short summary
We reprogrammed the latest WaterGAP model (2.2e) to create a sustainable global hydrological model. By utilizing best software practices like modular design, version control, and clear documentation, the new WaterGAP supports collaboration across teams. It can be easily understood, applied, and enhanced by both novice and experienced modellers. Additionally, we share the reprogramming process to assist in the reprogramming of other large geoscientific research software.
Seyed-Mohammad Hosseini-Moghari and Petra Döll
Hydrol. Earth Syst. Sci., 29, 4073–4092, https://doi.org/10.5194/hess-29-4073-2025, https://doi.org/10.5194/hess-29-4073-2025, 2025
Short summary
Short summary
Modeling reservoir outflow and storage is challenging due to limited publicly available data and human decision-making. For 100 reservoirs in the US, we examined how calibrating reservoir algorithms against outflow and storage-related variables affects performance. We found that calibration notably improves storage simulations, while outflow simulations are more influenced by the quality of inflow data. We recommend using remotely sensed storage anomalies to calibrate reservoir algorithms.
Katja Frieler, Stefan Lange, Jacob Schewe, Matthias Mengel, Simon Treu, Christian Otto, Jan Volkholz, Christopher P. O. Reyer, Stefanie Heinicke, Colin Jones, Julia L. Blanchard, Cheryl S. Harrison, Colleen M. Petrik, Tyler D. Eddy, Kelly Ortega-Cisneros, Camilla Novaglio, Ryan Heneghan, Derek P. Tittensor, Olivier Maury, Matthias Büchner, Thomas Vogt, Dánnell Quesada Chacón, Kerry Emanuel, Chia-Ying Lee, Suzana J. Camargo, Jonas Jägermeyr, Sam Rabin, Jochen Klar, Iliusi D. Vega del Valle, Lisa Novak, Inga J. Sauer, Gitta Lasslop, Sarah Chadburn, Eleanor Burke, Angela Gallego-Sala, Noah Smith, Jinfeng Chang, Stijn Hantson, Chantelle Burton, Anne Gädeke, Fang Li, Simon N. Gosling, Hannes Müller Schmied, Fred Hattermann, Thomas Hickler, Rafael Marcé, Don Pierson, Wim Thiery, Daniel Mercado-Bettín, Robert Ladwig, Ana Isabel Ayala-Zamora, Matthew Forrest, Michel Bechtold, Robert Reinecke, Inge de Graaf, Jed O. Kaplan, Alexander Koch, and Matthieu Lengaigne
EGUsphere, https://doi.org/10.5194/egusphere-2025-2103, https://doi.org/10.5194/egusphere-2025-2103, 2025
Short summary
Short summary
This paper describes the experiments and data sets necessary to run historic and future impact projections, and the underlying assumptions of future climate change as defined by the 3rd round of the ISIMIP Project (Inter-sectoral Impactmodel Intercomparison Project, isimip.org). ISIMIP provides a framework for cross-sectorally consistent climate impact simulations to contribute to a comprehensive and consistent picture of the world under different climate-change scenarios.
Robert Reinecke, Annemarie Bäthge, Ricarda Dietrich, Sebastian Gnann, Simon N. Gosling, Danielle Grogan, Andreas Hartmann, Stefan Kollet, Rohini Kumar, Richard Lammers, Sida Liu, Yan Liu, Nils Moosdorf, Bibi Naz, Sara Nazari, Chibuike Orazulike, Yadu Pokhrel, Jacob Schewe, Mikhail Smilovic, Maryna Strokal, Yoshihide Wada, Shan Zuidema, and Inge de Graaf
EGUsphere, https://doi.org/10.5194/egusphere-2025-1181, https://doi.org/10.5194/egusphere-2025-1181, 2025
Short summary
Short summary
Here we describe a collaborative effort to improve predictions of how climate change will affect groundwater. The ISIMIP groundwater sector combines multiple global groundwater models to capture a range of possible outcomes and reduce uncertainty. Initial comparisons reveal significant differences between models in key metrics like water table depth and recharge rates, highlighting the need for structured model intercomparisons.
Howlader Mohammad Mehedi Hasan, Petra Döll, Seyed-Mohammad Hosseini-Moghari, Fabrice Papa, and Andreas Güntner
Hydrol. Earth Syst. Sci., 29, 567–596, https://doi.org/10.5194/hess-29-567-2025, https://doi.org/10.5194/hess-29-567-2025, 2025
Short summary
Short summary
We calibrate a global hydrological model using multiple observations to analyse the benefits and trade-offs of multi-variable calibration. We found such an approach to be very important for understanding the real-world system. However, some observations are very essential to the system, in particular, streamflow. We also showed uncertainties in the calibration results, which are often useful for making informed decisions. We emphasize considering observation uncertainty in model calibration.
Hannes Müller Schmied, Tim Trautmann, Sebastian Ackermann, Denise Cáceres, Martina Flörke, Helena Gerdener, Ellen Kynast, Thedini Asali Peiris, Leonie Schiebener, Maike Schumacher, and Petra Döll
Geosci. Model Dev., 17, 8817–8852, https://doi.org/10.5194/gmd-17-8817-2024, https://doi.org/10.5194/gmd-17-8817-2024, 2024
Short summary
Short summary
Assessing water availability and water use at the global scale is challenging but essential for a range of purposes. We describe the newest version of the global hydrological model WaterGAP, which has been used for numerous water resource assessments since 1996. We show the effects of new model features, as well as model evaluations, against water abstraction statistics and observed streamflow and water storage anomalies. The publicly available model output for several variants is described.
Petra Döll, Howlader Mohammad Mehedi Hasan, Kerstin Schulze, Helena Gerdener, Lara Börger, Somayeh Shadkam, Sebastian Ackermann, Seyed-Mohammad Hosseini-Moghari, Hannes Müller Schmied, Andreas Güntner, and Jürgen Kusche
Hydrol. Earth Syst. Sci., 28, 2259–2295, https://doi.org/10.5194/hess-28-2259-2024, https://doi.org/10.5194/hess-28-2259-2024, 2024
Short summary
Short summary
Currently, global hydrological models do not benefit from observations of model output variables to reduce and quantify model output uncertainty. For the Mississippi River basin, we explored three approaches for using both streamflow and total water storage anomaly observations to adjust the parameter sets in a global hydrological model. We developed a method for considering the observation uncertainties to quantify the uncertainty of model output and provide recommendations.
Laura Müller and Petra Döll
Geosci. Commun., 7, 121–144, https://doi.org/10.5194/gc-7-121-2024, https://doi.org/10.5194/gc-7-121-2024, 2024
Short summary
Short summary
To be able to adapt to climate change, stakeholders need to be informed about future uncertain climate change hazards. Using freely available output of global hydrological models, we quantified future local changes in water resources and their uncertainty. To communicate these in participatory processes, we propose using "percentile boxes" to support the development of flexible strategies for climate risk management worldwide, involving stakeholders and scientists.
Thedini Asali Peiris and Petra Döll
Hydrol. Earth Syst. Sci., 27, 3663–3686, https://doi.org/10.5194/hess-27-3663-2023, https://doi.org/10.5194/hess-27-3663-2023, 2023
Short summary
Short summary
Hydrological models often overlook vegetation's response to CO2 and climate, impairing their ability to forecast impacts on evapotranspiration and water resources. To address this, we suggest involving two model variants: (1) the standard method and (2) a modified approach (proposed here) based on the Priestley–Taylor equation (PT-MA). While not universally applicable, a dual approach helps consider uncertainties related to vegetation responses to climate change, enhancing model representation.
Claudia Herbert and Petra Döll
Nat. Hazards Earth Syst. Sci., 23, 2111–2131, https://doi.org/10.5194/nhess-23-2111-2023, https://doi.org/10.5194/nhess-23-2111-2023, 2023
Short summary
Short summary
This paper presents a new method for selecting streamflow drought hazard indicators for monitoring drought hazard for human water supply and river ecosystems in large-scale drought early warning systems. Indicators are classified by their inherent assumptions about the habituation of people and ecosystems to the streamflow regime and their level of drought characterization, namely drought magnitude (water deficit at a certain point in time) and severity (cumulated magnitude since drought onset).
Martin Horwath, Benjamin D. Gutknecht, Anny Cazenave, Hindumathi Kulaiappan Palanisamy, Florence Marti, Ben Marzeion, Frank Paul, Raymond Le Bris, Anna E. Hogg, Inès Otosaka, Andrew Shepherd, Petra Döll, Denise Cáceres, Hannes Müller Schmied, Johnny A. Johannessen, Jan Even Øie Nilsen, Roshin P. Raj, René Forsberg, Louise Sandberg Sørensen, Valentina R. Barletta, Sebastian B. Simonsen, Per Knudsen, Ole Baltazar Andersen, Heidi Ranndal, Stine K. Rose, Christopher J. Merchant, Claire R. Macintosh, Karina von Schuckmann, Kristin Novotny, Andreas Groh, Marco Restano, and Jérôme Benveniste
Earth Syst. Sci. Data, 14, 411–447, https://doi.org/10.5194/essd-14-411-2022, https://doi.org/10.5194/essd-14-411-2022, 2022
Short summary
Short summary
Global mean sea-level change observed from 1993 to 2016 (mean rate of 3.05 mm yr−1) matches the combined effect of changes in water density (thermal expansion) and ocean mass. Ocean-mass change has been assessed through the contributions from glaciers, ice sheets, and land water storage or directly from satellite data since 2003. Our budget assessments of linear trends and monthly anomalies utilise new datasets and uncertainty characterisations developed within ESA's Climate Change Initiative.
Tom Gleeson, Thorsten Wagener, Petra Döll, Samuel C. Zipper, Charles West, Yoshihide Wada, Richard Taylor, Bridget Scanlon, Rafael Rosolem, Shams Rahman, Nurudeen Oshinlaja, Reed Maxwell, Min-Hui Lo, Hyungjun Kim, Mary Hill, Andreas Hartmann, Graham Fogg, James S. Famiglietti, Agnès Ducharne, Inge de Graaf, Mark Cuthbert, Laura Condon, Etienne Bresciani, and Marc F. P. Bierkens
Geosci. Model Dev., 14, 7545–7571, https://doi.org/10.5194/gmd-14-7545-2021, https://doi.org/10.5194/gmd-14-7545-2021, 2021
Short summary
Short summary
Groundwater is increasingly being included in large-scale (continental to global) land surface and hydrologic simulations. However, it is challenging to evaluate these simulations because groundwater is
hiddenunderground and thus hard to measure. We suggest using multiple complementary strategies to assess the performance of a model (
model evaluation).
Camelia-Eliza Telteu, Hannes Müller Schmied, Wim Thiery, Guoyong Leng, Peter Burek, Xingcai Liu, Julien Eric Stanislas Boulange, Lauren Seaby Andersen, Manolis Grillakis, Simon Newland Gosling, Yusuke Satoh, Oldrich Rakovec, Tobias Stacke, Jinfeng Chang, Niko Wanders, Harsh Lovekumar Shah, Tim Trautmann, Ganquan Mao, Naota Hanasaki, Aristeidis Koutroulis, Yadu Pokhrel, Luis Samaniego, Yoshihide Wada, Vimal Mishra, Junguo Liu, Petra Döll, Fang Zhao, Anne Gädeke, Sam S. Rabin, and Florian Herz
Geosci. Model Dev., 14, 3843–3878, https://doi.org/10.5194/gmd-14-3843-2021, https://doi.org/10.5194/gmd-14-3843-2021, 2021
Short summary
Short summary
We analyse water storage compartments, water flows, and human water use sectors included in 16 global water models that provide simulations for the Inter-Sectoral Impact Model Intercomparison Project phase 2b. We develop a standard writing style for the model equations. We conclude that even though hydrologic processes are often based on similar equations, in the end these equations have been adjusted, or the models have used different values for specific parameters or specific variables.
Eklavyya Popat and Petra Döll
Nat. Hazards Earth Syst. Sci., 21, 1337–1354, https://doi.org/10.5194/nhess-21-1337-2021, https://doi.org/10.5194/nhess-21-1337-2021, 2021
Short summary
Short summary
Two drought hazard indices are presented that combine drought deficit and anomaly aspects: one for soil moisture drought (SMDAI) where we simplified the DSI and the other for streamflow drought (QDAI), which is to our knowledge the first ever deficit anomaly drought index including surface water demand. Both indices are tested at the global scale with WaterGAP 2.2d outputs, providing more differentiated spatial and temporal patterns distinguishing the actual degree of respective drought hazard.
Hannes Müller Schmied, Denise Cáceres, Stephanie Eisner, Martina Flörke, Claudia Herbert, Christoph Niemann, Thedini Asali Peiris, Eklavyya Popat, Felix Theodor Portmann, Robert Reinecke, Maike Schumacher, Somayeh Shadkam, Camelia-Eliza Telteu, Tim Trautmann, and Petra Döll
Geosci. Model Dev., 14, 1037–1079, https://doi.org/10.5194/gmd-14-1037-2021, https://doi.org/10.5194/gmd-14-1037-2021, 2021
Short summary
Short summary
In a globalized world with large flows of virtual water between river basins and international responsibilities for the sustainable development of the Earth system and its inhabitants, quantitative estimates of water flows and storages and of water demand by humans are required. Global hydrological models such as WaterGAP are developed to provide this information. Here we present a thorough description, evaluation and application examples of the most recent model version, WaterGAP v2.2d.
Robert Reinecke, Hannes Müller Schmied, Tim Trautmann, Lauren Seaby Andersen, Peter Burek, Martina Flörke, Simon N. Gosling, Manolis Grillakis, Naota Hanasaki, Aristeidis Koutroulis, Yadu Pokhrel, Wim Thiery, Yoshihide Wada, Satoh Yusuke, and Petra Döll
Hydrol. Earth Syst. Sci., 25, 787–810, https://doi.org/10.5194/hess-25-787-2021, https://doi.org/10.5194/hess-25-787-2021, 2021
Short summary
Short summary
Billions of people rely on groundwater as an accessible source of drinking water and for irrigation, especially in times of drought. Groundwater recharge is the primary process of regenerating groundwater resources. We find that groundwater recharge will increase in northern Europe by about 19 % and decrease by 10 % in the Amazon with 3 °C global warming. In the Mediterranean, a 2 °C warming has already lead to a reduction in recharge by 38 %. However, these model predictions are uncertain.
Denise Cáceres, Ben Marzeion, Jan Hendrik Malles, Benjamin Daniel Gutknecht, Hannes Müller Schmied, and Petra Döll
Hydrol. Earth Syst. Sci., 24, 4831–4851, https://doi.org/10.5194/hess-24-4831-2020, https://doi.org/10.5194/hess-24-4831-2020, 2020
Short summary
Short summary
We analysed how and to which extent changes in water storage on continents had an effect on global ocean mass over the period 1948–2016. Continents lost water to oceans at an accelerated rate, inducing sea level rise. Shrinking glaciers explain 81 % of the long-term continental water mass loss, while declining groundwater levels, mainly due to sustained groundwater pumping for irrigation, is the second major driver. This long-term decline was partly offset by the impoundment of water in dams.
Cited articles
Abernathey, R. P., Augspurger, T., Banihirwe, A., Blackmon-Luca, C. C., Crone, T. J., Gentemann, C. L., Hamman, J. J., Henderson, N., Lepore, C., McCaie, T. A., Robinson, N. H., and Signell, R. P.: Cloud-Native Repositories for Big Scientific Data, Comput. Sci. Eng., 23, 26–35, https://doi.org/10.1109/MCSE.2021.3059437, 2021.
Alexander, K. and Easterbrook, S. M.: The software architecture of climate models: a graphical comparison of CMIP5 and EMICAR5 configurations, Geosci. Model Dev., 8, 1221–1232, https://doi.org/10.5194/gmd-8-1221-2015, 2015.
Anzt, H., Bach, F., Druskat, S., Löffler, F., Loewe, A., Renard, B., Seemann, G., Struck, A., Achhammer, E., Aggarwal, P., Appel, F., Bader, M., Brusch, L., Busse, C., Chourdakis, G., Dabrowski, P., Ebert, P., Flemisch, B., Friedl, S., Fritzsch, B., Funk, M., Gast, V., Goth, F., Grad, J., Hegewald, J., Hermann, S., Hohmann, F., Janosch, S., Kutra, D., Linxweiler, J., Muth, T., Peters-Kottig, W., Rack, F., Raters, F., Rave, S., Reina, G., Reißig, M., Ropinski, T., Schaarschmidt, J., Seibold, H., Thiele, J., Uekermann, B., Unger, S., and Weeber, R.: An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action [version 2; peer review: 2 approved], F1000Research, 9, 295, https://doi.org/10.12688/f1000research.23224.2, 2021.
Arafat, O. and Riehle, D.: The comment density of open source software code, in: 2009 31st International Conference on Software Engineering – Companion Volume, 16–24 May 2009, Vancouver, BC, Canada, 195–198, https://doi.org/10.1109/ICSE-COMPANION.2009.5070980, 2009.
Azmi, E., Ehret, U., Weijs, S. V., Ruddell, B. L., and Perdigão, R. A. P.: Technical note: “Bit by bit”: a practical and general approach for evaluating model computational complexity vs. model performance, Hydrol. Earth Syst. Sci., 25, 1103–1115, https://doi.org/10.5194/hess-25-1103-2021, 2021.
Barker, M., Chue Hong, N. P., Katz, D. S., Lamprecht, A.-L., Martinez-Ortiz, C., Psomopoulos, F., Harrow, J., Castro, L. J., Gruenpeter, M., Martinez, P. A., and Honeyman, T.: Introducing the FAIR Principles for research software, Sci. Data, 9, 622, https://doi.org/10.1038/s41597-022-01710-x, 2022.
Barton, C. M., Lee, A., Janssen, M. A., van der Leeuw, S., Tucker, G. E., Porter, C., Greenberg, J., Swantek, L., Frank, K., Chen, M., and Jagers, H. R. A.: How to make models more useful, P. Natl. Acad. Sci. USA, 119, e2202112119, https://doi.org/10.1073/pnas.2202112119, 2022.
Boehm, B. W.: Software engineering economics, Prentice-Hall, Englewood Cliffs, NJ, 57–96, ISBN 0138221227, 1981.
Boyter, B.: boyter/scc, GitHub [code], https://github.com/boyter/scc, last access: 3 March 2024.
Burek, P., Satoh, Y., Kahil, T., Tang, T., Greve, P., Smilovic, M., Guillaumot, L., Zhao, F., and Wada, Y.: Development of the Community Water Model (CWatM v1.04) – a high-resolution hydrological model for global and regional assessment of integrated water resources management, Geosci. Model Dev., 13, 3267–3298, https://doi.org/10.5194/gmd-13-3267-2020, 2020.
Capiluppi, A., Boldyreff, C., Beecher, K., and Adams, P. J.: Quality Factors and Coding Standards – a Comparison Between Open Source Forges, Electronic Notes in Theoretical Computer Science, 233, 89–103, https://doi.org/10.1016/j.entcs.2009.02.063, 2009.
Carver, J., Heaton, D., Hochstein, L., and Bartlett, R.: Self-Perceptions about Software Engineering: A Survey of Scientists and Engineers, Comput. Sci. Eng., 15, 7–11, https://doi.org/10.1109/MCSE.2013.12, 2013.
Carver, J. C., Weber, N., Ram, K., Gesing, S., and Katz, D. S.: A survey of the state of the practice for research software in the United States, PeerJ Computer Science, 8, e963, https://doi.org/10.7717/peerj-cs.963, 2022.
Colazo, J. and Fang, Y.: Impact of license choice on Open Source Software development activity, J. Am. Soc. Inf. Sci. Tec., 60, 997–1011, https://doi.org/10.1002/asi.21039, 2009.
Döll, P., Sester, M., Feuerhake, U., Frahm, H., Fritzsch, B., Hezel, D. C., Kaus, B., Kolditz, O., Linxweiler, J., Müller Schmied, H., Nyenah, E., Risse, B., Schielein, U., Schlauch, T., Streck, T., and van den Oord, G.: Sustainable research software for high-quality computational research in the Earth System Sciences: Recommendations for universities, funders and the scientific community in Germany, https://doi.org/10.23689/fidgeo-5805, 2023.
Editorial: Does your code stand up to scrutiny?, Nature, 555, 142–142, https://doi.org/10.1038/d41586-018-02741-4, 2018.
Editorial: Giving software its due, Nat. Methods, 16, 207–207, https://doi.org/10.1038/s41592-019-0350-x, 2019.
Fowler, M.: Refactoring, 2nd edn., Addison Wesley, Boston, MA, ISBN 0134757599, 2019.
Frieler, K. and Vega, I.: ISIMIP & ISIpedia – Inter-sectoral impact modeling and communication of national impact assessments, Bonn Climate Change Conference, 19 June 2019, session SBSTA 50, https://unfccc.int/documents/197148 (2 December 2024), 2019.
Gamma, E., Helm, R., Johnson, R., and Vlissides, J.: Design patterns, Addison Wesley, Boston, MA, ISBN 0201633612, 1994.
Guaman, D., Delgado, S., and Perez, J.: Classifying Model-View-Controller Software Applications Using Self-Organizing Maps, IEEE Access, 9, 45201–45229, https://doi.org/10.1109/ACCESS.2021.3066348, 2021.
Hannay, J. E., MacLeod, C., Singer, J., Langtangen, H. P., Pfahl, D., and Wilson, G.: How do scientists develop and use scientific software?, in: 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering, 23 May 2009, Vancouver, BC, Canada, 1–8, https://doi.org/10.1109/SECSE.2009.5069155, 2009.
Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M. H., Brett, M., Haldane, A., del Río, J. F., Wiebe, M., Peterson, P., Gérard-Marchant, P., Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H., Gohlke, C., and Oliphant, T. E.: Array programming with NumPy, Nature, 585, 357–362, https://doi.org/10.1038/s41586-020-2649-2, 2020.
He, H.: Understanding Source Code Comments at Large-Scale, in: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 26–30 August 2019, Tallinn, Estonia, 1217–1219, https://doi.org/10.1145/3338906.3342494, 2019.
Hofmann, H., Wickham, H., and Kafadar, K.: Letter-Value Plots: Boxplots for Large Data, J. Comput. Graph. Stat., 26, 469–477, https://doi.org/10.1080/10618600.2017.1305277, 2017.
ISIMIP: https://www.isimip.org/, last access: 23 March 2024.
Jay, C. and Haines, R.: Reproducible and Sustainable Research Software, in: Web Accessibility: A Foundation for Research, edited by: Yesilada, Y. and Harper, S., Springer, London, 211–221, https://doi.org/10.1007/978-1-4471-7440-0_12, 2019.
Jiménez, R. C., Kuzak, M., Alhamdoosh, M., Barker, M., Batut, B., Borg, M., Capella-Gutierrez, S., Hong, N. C., Cook, M., Corpas, M., Flannery, M., Garcia, L., Gelpí, J. L., Gladman, S., Goble, C., Ferreiro, M. G., Gonzalez-Beltran, A., Griffin, P. C., Grüning, B., Hagberg, J., Holub, P., Hooft, R., Ison, J., Katz, D. S., Leskošek, B., Gómez, F. L., Oliveira, L. J., Mellor, D., Mosbergen, R., Mulder, N., Perez-Riverol, Y., Pergl, R., Pichler, H., Pope, B., Sanz, F., Schneider, M. V., Stodden, V., Suchecki, R., Vařeková, R. S., Talvik, H.-A., Todorov, I., Treloar, A., Tyagi, S., van Gompel, M., Vaughan, D., Via, A., Wang, X., Watson-Haigh, N. S., and Crouch, S.: Four simple recommendations to encourage best practices in research software, https://doi.org/10.12688/f1000research.11407.1, 13 June 2017.
JuliaReachDevDocs: https://juliareach.github.io/JuliaReachDev
Docs/latest/guidelines/, last access: 11 September 2024.
Docs/latest/guidelines/, last access: 11 September 2024.
Katz, D. S.: Research Software: Challenges & Actions. The Future of Research Software: International Funders Workshop, Amsterdam, the Netherlands, https://doi.org/10.5281/zenodo.7295423, 2022.
Kemp, L., Xu, C., Depledge, J., Ebi, K. L., Gibbins, G., Kohler, T. A., Rockström, J., Scheffer, M., Schellnhuber, H. J., Steffen, W., and Lenton, T. M.: Climate Endgame: Exploring catastrophic climate change scenarios, P. Natl. Acad. Sci. USA, 119, e2108146119, https://doi.org/10.1073/pnas.2108146119, 2022.
Long, J.: Understanding the Role of Core Developers in Open Source Development, Journal of Information, Information Technology, and Organizations (Years 1–3), 1, 075–085, 2006.
McConnell, S.: A Practical Handbook of Software Construction, in: Code Complete, 2nd edn., Microsoft Press, USA, 565–596, ISBN 0735619670, 2004.
McKiernan, E. C., Barba, L., Bourne, P. E., Carter, C., Chandler, Z., Choudhury, S., Jacobs, S., Katz, D. S., Lieggi, S., Plale, B., and Tananbaum, G.: Policy recommendations to ensure that research software is openly accessible and reusable, PLOS Biol., 21, 1–4, https://doi.org/10.1371/journal.pbio.3002204, 2023.
Merow, C., Boyle, B., Enquist, B. J., Feng, X., Kass, J. M., Maitner, B. S., McGill, B., Owens, H., Park, D. S., Paz, A., Pinilla-Buitrago, G. E., Urban, M. C., Varela, S., and Wilson, A. M.: Better incentives are needed to reward academic software development, Nat. Ecol. Evol., 7, 626–627, https://doi.org/10.1038/s41559-023-02008-w, 2023.
Molnar, A.-J., Motogna, S., and Vlad, C.: Using static analysis tools to assist student project evaluation, in: Proceedings of the 2nd ACM SIGSOFT International Workshop on Education through Advanced Software Engineering and Artificial Intelligence, ESEC/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual, 9 November 2020, USA, 7–12, https://doi.org/10.1145/3412453.3423195, 2020.
Müller Schmied, H., Trautmann, T., Ackermann, S., Cáceres, D., Flörke, M., Gerdener, H., Kynast, E., Peiris, T. A., Schiebener, L., Schumacher, M., and Döll, P.: The global water resources and use model WaterGAP v2.2e: description and evaluation of modifications and new features, Geosci. Model Dev. Discuss. [preprint], https://doi.org/10.5194/gmd-2023-213, in review, 2023.
Nangia, U. and Katz, D. S.: Track 1 Paper: Surveying the U. S. National Postdoctoral Association Regarding Software Use and Training in Research, https://doi.org/10.6084/m9.figshare.5328442.v3, 2017.
Nüst, D., Sochat, V., Marwick, B., Eglen, S. J., Head, T., Hirst, T., and Evans, B. D.: Ten simple rules for writing Dockerfiles for reproducible data science, PLoS Comput. Biol., 16, e1008316, https://doi.org/10.1371/journal.pcbi.1008316, 2020.
Nyenah, E., Reinecke, R., and Döll, P.: Towards a sustainable utilization of the global hydrological research software WaterGAP, EGU General Assembly 2023, Vienna, Austria, 24–28 Apr 2023, EGU23-4453, https://doi.org/10.5194/egusphere-egu23-4453, 2023.
Nyenah, E., Döll, P., Katz, D. S., and Reinecke, R.: Software sustainability of global impact models (Dataset and analysis script), Zenodo [data set], https://doi.org/10.5281/zenodo.10245636, 2024.
Obermüller, F., Bloch, L., Greifenstein, L., Heuer, U., and Fraser, G.: Code Perfumes: Reporting Good Code to Encourage Learners, in: The 16th Workshop in Primary and Secondary Computing Education, WiPSCE '21: The 16th Workshop in Primary and Secondary Computing Education, Virtual Event, 18–20 October 2021, Germany, 1–10, https://doi.org/10.1145/3481312.3481346, 2021.
Plösch, R., Bräuer, J., Körner, C., and Saft, M.: Measuring, Assessing and Improving Software Quality based on Object-Oriented Design Principles, Open Computer Science, 6, 187–207, https://doi.org/10.1515/comp-2016-0016, 2016.
Prinn, R. G.: Development and application of earth system models, P. Natl. Acad. Sci. USA, 110, 3673–3680, https://doi.org/10.1073/pnas.1107470109, 2013.
Rashid, M., Clarke, P. M., and O'Connor, R. V.: A systematic examination of knowledge loss in open source software projects, Int. J. Inform. Manage., 46, 104–123, https://doi.org/10.1016/j.ijinfomgt.2018.11.015, 2019.
Reinecke, R., Trautmann, T., Wagener, T., and Schüler, K.: The critical need to foster computational reproducibility, Environ. Res. Lett., 17, 4, https://doi.org/10.1088/1748-9326/ac5cf8, 2022.
Research Software Alliance: Amsterdam Declaration on Funding Research Software Sustainability, https://doi.org/10.5281/ZENODO.8325436, 2023.
Sachan, R. K., Nigam, A., Singh, A., Singh, S., Choudhary, M., Tiwari, A., and Kushwaha, D. S.: Optimizing Basic COCOMO Model Using Simplified Genetic Algorithm, Procedia Comput. Sci., 89, 492–498, https://doi.org/10.1016/j.procs.2016.06.107, 2016.
Sarkar, S., Kak, A. C., and Rama, G. M.: Metrics for Measuring the Quality of Modularization of Large-Scale Object-Oriented Software, IEEE T. Software Eng., 34, 700–720, https://doi.org/10.1109/TSE.2008.43, 2008.
Satoh, Y., Yoshimura, K., Pokhrel, Y., Kim, H., Shiogama, H., Yokohata, T., Hanasaki, N., Wada, Y., Burek, P., Byers, E., Schmied, H. M., Gerten, D., Ostberg, S., Gosling, S. N., Boulange, J. E. S., and Oki, T.: The timing of unprecedented hydrological drought under climate change, Nat. Commun., 13, 3287, https://doi.org/10.1038/s41467-022-30729-2, 2022.
Sauer, I. J., Reese, R., Otto, C., Geiger, T., Willner, S. N., Guillod, B. P., Bresch, D. N., and Frieler, K.: Climate signals in river flood damages emerge under sound regional disaggregation, Nat. Commun., 12, 2128, https://doi.org/10.1038/s41467-021-22153-9, 2021.
Schmidhuber, J. and Tubiello, F. N.: Global food security under climate change, P. Natl. Acad. Sci. USA, 104, 19703–19708, https://doi.org/10.1073/pnas.0701976104, 2007.
Simmons, A. J., Barnett, S., Rivera-Villicana, J., Bajaj, A., and Vasa, R.: A large-scale comparative analysis of Coding Standard conformance in Open-Source Data Science projects, in: Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), ESEM '20: ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, 5–9 October 2020, Bari, Italy, 1–11, https://doi.org/10.1145/3382494.3410680, 2020.
SLOCCount: https://stuff.mit.edu/iap/debian/solutions/sloccount-2.26/sloccount.html, last access: 4 March 2024.
Stacke, T. and Hagemann, S.: HydroPy (v1.0): a new global hydrology model written in Python, Geosci. Model Dev., 14, 7795–7816, https://doi.org/10.5194/gmd-14-7795-2021, 2021a.
Stacke, T. and Hagemann, S.: Source code for the global hydrological model HydroPy, Zenodo [code], https://doi.org/10.5281/zenodo.4541381, 2021b.
Ståhl, D. and Bosch, J.: Modeling continuous integration practice differences in industry software development, J. Syst. Software, 87, 48–59, https://doi.org/10.1016/j.jss.2013.08.032, 2014.
Stamelos, I., Angelis, L., Oikonomou, A., and Bleris, G. L.: Code quality analysis in open source software development, Inform. Syst. J., 12, 43–60, https://doi.org/10.1046/j.1365-2575.2002.00117.x, 2002.
Trisovic, A., Lau, M. K., Pasquier, T., and Crosas, M.: A large-scale study on research code quality and execution, Sci. Data, 9, 60, https://doi.org/10.1038/s41597-022-01143-6, 2022.
Turk, D., Robert, F., and Rumpe, B.: Assumptions Underlying Agile Software-Development Processes, J. Database Manage., 16, 62–87, https://doi.org/10.4018/jdm.2005100104, 2005.
van Eeuwijk, S., Bakker, T., Cruz, M., Sarkol, V., Vreede, B., Aben, B., Aerts, P., Coen, G., van Dijk, B., Hinrich, P., Karvovskaya, L., Ruijter, M. K., Koster, J., Maassen, J., Roelofs, M., Rijnders, J., Schroten, A., Sesink, L., van der Togt, C., Vinju, J., and de Willigen, P.: Research software sustainability in the Netherlands: Current practices and recommendations, Zenodo, https://doi.org/10.5281/zenodo.4543569, 2021.
Van Snyder, W.: Scientific Programming in Fortran, Scientific Programming, 15, 3–8, https://doi.org/10.1155/2007/930816, 2007.
Wagener, T., Gleeson, T., Coxon, G., Hartmann, A., Howden, N., Pianosi, F., Rahman, M., Rosolem, R., Stein, L., and Woods, R.: On doing hydrology with dragons: Realizing the value of perceptual models and knowledge accumulation, WIREs Water, 8, e1550, https://doi.org/10.1002/wat2.1550, 2021.
Wan, W., Döll, P., and Zheng, H.: Risk of Climate Change for Hydroelectricity Production in China Is Small but Significant Reductions Cannot Be Precluded for More Than a Third of the Installed Capacity, Water Resour. Res., 58, e2022WR032380, https://doi.org/10.1029/2022WR032380, 2022.
Wang, Y., Zheng, B., and Huang, H.: Complying with Coding Standards or Retaining Programming Style: A Quality Outlook at Source Code Level, Journal of Software Engineering and Applications, 1, 88–91, https://doi.org/10.4236/jsea.2008.11013, 2008.
Warszawski, L., Frieler, K., Huber, V., Piontek, F., Serdeczny, O., and Schewe, J.: The Inter-Sectoral Impact Model Intercomparison Project (ISI–MIP): Project framework, P. Natl. Acad. Sci. USA, 111, 3228–3232, https://doi.org/10.1073/pnas.1312330110, 2014.
Wijendra, D. R. and Hewagamage, K. P.: Software Complexity Reduction through the Process Automation in Software Development Life Cycle, in: 2021 Fourth International Conference on Electrical, Computer and Communication Technologies (ICECCT), 15–17 September 2021, Erode, India, 1–7, https://doi.org/10.1109/ICECCT52121.2021.9616781, 2021.
Wilson, G., Aruliah, D. A., Brown, C. T., Hong, N. P. C., Davis, M., Guy, R. T., Haddock, S. H. D., Huff, K. D., Mitchell, I. M., Plumbley, M. D., Waugh, B., White, E. P., and Wilson, P.: Best Practices for Scientific Computing, PLOS Biol., 12, e1001745, https://doi.org/10.1371/journal.pbio.1001745, 2014.
Zhou, N., Zhou, H., and Hoppe, D.: Containerisation for High Performance Computing Systems: Survey and Prospects, IEEE T. Software Eng., 49, 2722–2740, https://doi.org/10.1109/TSE.2022.3229221, 2023.
Short summary
Research software is vital for scientific progress but is often developed by scientists with limited skills, time, and funding, leading to challenges in usability and maintenance. Our study across 10 sectors shows strengths in version control, open-source licensing, and documentation while emphasizing the need for containerization and code quality. We recommend workshops; code quality metrics; funding; and following the findable, accessible, interoperable, and reusable (FAIR) standards.
Research software is vital for scientific progress but is often developed by scientists with...
