The Variable Infiltration Capacity model version 5 (VIC-5): infrastructure improvements for new applications and reproducibility

The Variable Infiltration Capacity (VIC) model is a macroscale semi-distributed hydrologic model. VIC development began in the early 1990s and the model has since been used extensively for basinto global-scale applications that include hydrologic dataset construction, trend analysis of hydrologic fluxes and states, data evaluation and assimilation, forecasting, coupled climate modeling, and climate change impact assessment. Ongoing operational applications of the VIC model include the University of Washington’s drought monitoring and forecasting systems and NASA’s Land Data Assimilation System. This paper documents the development of VIC version 5 (VIC-5), which includes a major reconfiguration of the legacy VIC source code to support a wider range of modern hydrologic modeling applications. The VIC source code has been moved to a public GitHub repository to encourage participation by the broader user and developer communities. The reconfiguration has separated the core physics of the model from the driver source code, whereby the latter is responsible for memory allocation, preprocessing and post-processing, and input–output (I–O). VIC-5 includes four drivers that use the same core physics modules, but which allow for different methods for accessing this core to enable different model applications. Finally, VIC-5 is distributed with robust test infrastructure, components of which routinely run during development using cloud-hosted continuous integration. The work described here provides an example to the model development community for extending the life of a legacy model that is being used extensively. The development and release of VIC-5 represents a significant step forward for the VIC user community in terms of support for existing and new model applications, reproducibility, and scientific robustness.

Abstract.The Variable Infiltration Capacity (VIC) model is a macroscale semi-distributed hydrologic model.VIC development began in the early 1990s and the model has since been used extensively for basin-to global-scale applications that include hydrologic dataset construction, trend analysis of hydrologic fluxes and states, data evaluation and assimilation, forecasting, coupled climate modeling, and climate change impact assessment.Ongoing operational applications of the VIC model include the University of Washington's drought monitoring and forecasting systems and NASA's Land Data Assimilation System.This paper documents the development of VIC version 5 (VIC-5), which includes a major reconfiguration of the legacy VIC source code to support a wider range of modern hydrologic modeling applications.The VIC source code has been moved to a public GitHub repository to encourage participation by the broader user and developer communities.The reconfiguration has separated the core physics of the model from the driver source code, whereby the latter is responsible for memory allocation, preprocessing and post-processing, and input-output (I-O).VIC-5 includes four drivers that use the same core physics modules, but which allow for different methods for accessing this core to enable different model applications.Finally, VIC-5 is distributed with robust test infrastructure, components of which routinely run during development using cloud-hosted continuous integration.The work described here provides an example to the model development community for extending the life of a legacy model that is being used extensively.The development and release of VIC-5 rep-resents a significant step forward for the VIC user community in terms of support for existing and new model applications, reproducibility, and scientific robustness.

Introduction
The Variable Infiltration Capacity (VIC) model (Liang et al., 1994) is a semi-distributed macroscale hydrologic model (MHM) that has been applied in a broad set of use cases.The model has been used for numerous water and energy balance studies in the US (Abdulla and Lettenmaier, 1997;Nijssen et al., 1997), the Arctic (Adam and Lettenmaier, 2008;Su et al., 2005;Tan et al., 2011;Hamman et al., 2016), and globally (Nijssen et al., 2001a, b, c;Sheffield et al., 2009).One result of these studies has been refinement of the model to better represent key hydrological processes (Andreadis et al., 2009;Cherkauer et al., 2003;Liang et al., 1996Liang et al., , 1999)).Ongoing real-time applications of the VIC model include the University of California, Los Angeles' drought monitoring and forecasting systems (http://www.hydro.ucla.edu/SurfaceWaterGroup/forecast/monitor/index.shtml, last access: 20 August 2018), and NASA's Land Data Assimilation System (LDAS; https://ldas.gsfc.nasa.gov,last access: 20 August 2018).Table 1 provides additional examples of the use of VIC for hydrological, water resources, and water and energy budget applications.
Published by Copernicus Publications on behalf of the European Geosciences Union.Hamman et al. (2017) Historic trend analysis Snow Hamlet et al. (2005), Shi et al. (2011), Mote et al. (2005) Land use and/or land cover change Matheussen et al. (2000) Streamflow Hamlet and Lettenmaier (2007), Hamlet et al. (2007), Tan et al. (2011) Drought Gao et al. (2011), Sheffield andWood (2007, 2008), Sheffield et al. (2009), Wang et al. (2011), Nijssen et al. (2014) Although the motivation for the development of VIC, as documented in the original VIC publication (Liang et al., 1994), was as a land surface scheme for coupled landatmosphere models and earth system models (ESMs), the VIC model has been applied predominantly in uncoupled modeling studies in which there is no feedback from the land surface to the atmosphere.VIC and other uncoupled MHMs have continued to exist alongside fully coupled land surface schemes and have developed a large user community for a number of reasons: (a) their focus on the representation of hydrological processes often resulted in simulations that were hydrologically more credible than the land surface schemes typically used in coupled land-atmosphere models (as shown, for instance, in the Project for Intercomparison of Land Surface Parameterizations, PILPS; Bowling et al., 2003;Wood et al., 1998); (b) their ability to simulate variables that are directly relevant for water resources management, such as streamflow, makes them useful for planning and impact studies; and (c) their lower demand for compu-tational resources (compared with coupled models) allows research groups that do not have access to supercomputers to perform large-scale simulations.Most of these uncoupled MHMs can, for most applications, be run on single processors or small computer clusters.
Because VIC was traditionally operated as a stand-alone land surface scheme, there were no requirements for the model code to communicate with other model components in a coupled environment.In other words, there was no requirement for the model to "play well with others".Instead, the model run-time environment focused on simplicity with the goal that VIC could run without the need for specific machine architectures, compilers, or third-party libraries.Model development prior to version 5 did not follow a well-defined development track and was directed primarily by the needs of specific model experiments.Relatively little attention was paid to improvements of the model infrastructure, which had remained essentially unchanged since its initial release in the early 1990s, including input-output (I-O) formats.As a consequence, model implementation typically required extensive preprocessing and post-processing, including reformatting and processing of model input and output files.The few VIC applications within multi-model frameworks (e.g., NASA LIS; Kumar et al., 2006) required significant customized reconfiguration of the model interface.While VIC model source code was version controlled and publicly available since the early 1990s, model development was not always transparent.The source code repository of record resided on a private machine at the University of Washington, where most of the development was conducted.
Although the ability to run MHMs such as VIC in uncoupled mode has been one of their strengths, earth system science has reached a point at which the inability of some MHMs to be used in both coupled and uncoupled mode is a serious shortcoming.While a few previous efforts have been made to couple VIC to both atmospheric and fully coupled earth system models (e.g., Zhu et al., 2009;Hamman et al., 2016), these methods have been ad hoc and required significant rewriting of the model infrastructure.In effect, this resulted in a version of the model code that was coupled, but which would rapidly become obsolete relative to the uncoupled VIC version, which was the main path for model enhancement and improvement.
Interest in and advocacy for the use of established software development methods has grown in recent years in the fields of hydrology and the earth sciences to promote reproducibility of model and analysis results (e.g., Wilson et al., 2014;Ceola et al., 2015;Fienen and Bakker, 2016;Gil et al., 2016;Hutton et al., 2016).Opinions differ with respect to the best workflows to promote reproducibility.For example, an active discussion ensued in response to the paper by Hutton et al. (2016), not so much about the need for improved workflows, but about implementation details and the required level of rigor (Anel, 2017;Melsen et al., 2017;Hut et al., 2017;Hutton et al., 2017a, b).This debate takes place within a broader academic discussion of the need for open-source computer programs to increase scientific transparency in the model development and application process (Ince et al., 2012).Improved model source code maintenance, standardized test setups, and transparency comprise only part of this drive towards greater reproducibility.Other aspects include model and workflow documentation, versioning of datasets, and inclusion and standardization of metadata.It is with these points in mind that many of the new process and infrastructure improvements discussed in this paper were developed.
This paper documents the development of VIC version 5, hereafter referred to as VIC-5, a new major release of the VIC model.The motivation for the development of VIC-5 is to extend the usefulness of VIC by adding the necessary infrastructure to allow the model to operate in a number of coupled and uncoupled modes, while relying on the same underlying physics implementations, and to ensure that upgrades to the physics routines are automatically available to all model configurations.In this paper we describe the soft-ware, infrastructure, and associated improvements in VIC-5 and provide baseline model diagnostics in terms of computational performance.The initial VIC-5 release does not add new or improved physical process representations compared to the last release of the VIC-4 development track (version 4.2.d).Instead, the development of VIC-5 focused on reconfiguring the legacy VIC source code to support a wider range of modeling applications, to better integrate VIC with other applications, and to enhance reproducibility and source code maintenance.Therefore, this paper does not attempt to provide a scientific benchmarking of the VIC model.We have, however, provided a summary table of the key improvements to the VIC model since the original Liang et al. (1994) paper in Appendix A.
This paper is organized as follows.Section 2 briefly outlines the VIC legacy infrastructure in the model versions before VIC-5, details our design objectives, and discusses the major infrastructure changes implemented to meet those objectives.Section 3 includes results from test simulations with the new model code.The results and discussion from these test simulations are presented in Sect. 4. Concluding remarks are provided in Sect. 5 along with a discussion of our efforts to create and cultivate a VIC user community and to share the model source code.

Legacy implementation
Although VIC was originally intended to function as a land surface component in ESMs (Liang et al., 1994), its early development as a stand-alone model led to a model infrastructure that was poorly suited to coupling with atmospheric models.Most importantly, because model physics did not account for fluxes between adjacent grid cells (as was and remains common among land surface components of ESMs), VIC used a time-before-space computation order in which all time steps for a given model grid cell were completed before the model moved to the next location.We will refer to the time-before-space configuration as vector mode.This scheme is in direct conflict with atmospheric models, which use a space-before-time computation order in which all grid cells in the entire model domain are simulated for a single time step before advancing to the next time step.We will refer to the space-before-time configuration as image mode.In addition, consistent with the vector mode implementation, VIC's inputs and outputs consisted of one or more (ASCII or binary) files per grid cell, each containing the entire time series of a cell's fluxes and states.While this kept VIC's memory requirements relatively low, it led to an unwieldy number of files as domain size and grid resolution increased.Furthermore, analyzing, plotting, and publishing results from these files often required a time-consuming conversion to more flexible formats such as Unidata's Network www.geosci-model-dev.net/11/3481/2018/Geosci.Model Dev., 11, 3481-3496, 2018 Common Data Form (NetCDF; Rew and Davis, 1990).The challenge presented by separate files for each model grid cell is insidious.While it makes it easy to examine time series for individual grid cells, it makes it more difficult to examine time series of domain-wide model output and can hide errors in spatial model inputs and outputs that may be readily apparent upon inspection of a time series of spatial output.While VIC's original infrastructure may have made coupling within ESMs difficult, it also contributed to the model's wide uptake throughout the hydrologic modeling community.Previous versions of VIC were very portable, required no third-party libraries, were easy to run in single-point mode for testing or comparison with point observations, and were trivial to parallelize across many processors and machines by manually subdividing the domain and running each subdomain on a separate processor.It was this simple infrastructure, combined with the inclusion of built-in convenience tools such as the meteorological variable disaggregator based on the Mountain Climate Simulator (MT-CLIM) algorithms (Thornton and Running, 1999;Bohn et al., 2013), publicly available regional and global forcing and parameter datasets, and publicly available source code, that greatly contributed to VIC's wide adoption in the hydrologic modeling community.

Design goals for VIC-5
The VIC-5 release includes a complete refactor of the model source code.The design goals of the refactor included the following: separation of the model physical core from the driver code; multiple model drivers to support a range of modeling applications; parallel processing to facilitate efficient large domain simulations; machine-independent and self-documenting I-O formats such as NetCDF; model extensions framework to facilitate coupling of sub-model components; comprehensive test suite to guide future model use and development; source code availability and version control via an online platform to enable community participation in model maintenance and development and to promote transparency; and improved model documentation.
The implementation details for each of these topics are discussed in the following sections.A practical motivating factor in the development of VIC-5 was to enable coupling of VIC within the Regional Arctic System Model (RASM; Hamman et al., 2016), a regional implementation of the Community Earth System Model (CESM; Hurrell et al., 2013), while maintaining usability as a stand-alone model.In RASM, VIC-5 is directly coupled to CESM's flux coupler (CPL7; Craig et al., 2012) via the Model Coupling Toolkit interface (MCT; Larson et al., 2005).As implemented within RASM, this interface allows VIC to be coupled to the Weather Research and Forecasting (WRF) atmospheric model (Skamarock, 2008) and the RVIC streamflow routing model (Hamman et al., 2017) components, while maintaining the legacy and stand-alone model infrastructure, which have a large existing user base.The successful coupling of VIC-5 within RASM is a key demonstration of the benefits of the source code refactor.The new infrastructure will facilitate VIC to be coupled more readily to other modeling systems in the future.

Drivers
The most significant change in the model reconfiguration was the separation of the physical model core from the model drivers, which are responsible for memory allocation, preprocessing and post-processing, and input and output (I-O).The VIC-5 source code structure is depicted in Fig. 1.The initial VIC-5 release includes four drivers, each using the same physical model core.
1.The classic driver supports legacy VIC configurations and runs in the traditional time-before-space or vector mode.It uses ASCII and flat-binary file formats for I-O.With the exceptions of the removal of the forcing disaggregator and minor changes to the formatting and naming of input options, the classic driver is backwards compatible with VIC version 4.2.d.More discussion on the removal of the forcing disaggregator from the classic driver is provided in Sect.2.6.
2. The image driver includes a stand-alone space-beforetime configuration, NetCDF I-O, and uses the message passing interface (MPI; Gropp et al., 1996) and OpenMP (Dagum and Menon, 1998)  Additionally, VIC includes a set of general "shared driver utilities" that may be used by the individual drivers.These utilities perform driver-level tasks, such as logging and allocating memory, that are shared among all drivers.Grouping these common utilities eliminated the duplication of source code among the individual drivers.

Extensions
In VIC-5, we have introduced a model extensions framework to facilitate the coupling of sub-model components.Previous versions of VIC have added extended functionality to the model such as streamflow routing (Lohmann et al., 1996;Hamman et al., 2017), reservoirs, irrigation (Haddeland et al., 2006), glaciers and ice sheets, and dynamic vegetation and crops (Adam et al., 2015).These extensions were added either as stand-alone post-processing options or via highly customized interfaces.The motivating factor for developing a model extensions framework was to allow for the optional coupling of existing sub-models that are beyond the scope of core VIC applications or that are not feasible to add for all drivers.
The updated model configuration facilitates the direct coupling of new and existing subcomponents to VIC.Many of the previously developed sub-model components required information from neighboring grid cells or upstream processes (e.g., reservoir operations, irrigation).Leveraging the new functionality within the image driver, VIC is now able to provide information on storages and states from anywhere in the model domain to these sub-models.As of VIC version 5.1, only the streamflow routing extension, which couples the RVIC streamflow routing model (Hamman et al., 2017) to the image driver, is complete.
A key concept in the design of the VIC extensions framework is to allow for modular use and development of specific extensions.Therefore, each extension is enabled via compiletime options with the requirement that the model be able to run with or without the extension turned on.When turned off, each extension provides stub or no operation functions that allow the model to run without impacting the simulation.When turned on, each extension may use or modify state and flux variables from the core VIC model.

Parallel computing
The VIC model does not include horizontal exchange between grid cells.In the past, this feature has allowed users to manually parallelize their VIC applications by running VIC simulations for different grid cells on separate cores or computers (a technique referred to as poor man's parallelization).While this approach generally worked well for increasing the throughput of model simulations, it required laborious and time-intensive post-processing of the model output.Oftentimes, the time required to post-process these VIC simulations exceeded the model run time.In the image and CESM drivers in VIC-5, we have added formal parallel processing support by utilizing the MPI standard and shared memory threading via OpenMP.The parallel scaling performance of the VIC-5 image driver is discussed in detail in Sect.3.
Because VIC model grid cells do not exchange water or energy with their neighbors, each grid cell can be modeled independently.Our MPI parallelization strategy is to use MPI only to facilitate I-O operations.Thus, all I-O is performed by the master process.Inputs are read and scattered to the individual processors, while outputs are gathered and written out by the master process.The default domain decomposition uses a simple round-robin approach in which each grid cell in the model domain is sequentially distributed to another processor.For example, in the case of four MPI processes and 20 model grid cells, the first MPI process would handle grid cells 1, 5, 9, 13, and 17; the second would handle 2, 6, 10, 14, and 18 and so forth.In practice, this ensures a similar computational load for each MPI process for the most common VIC spatial configuration (a regular 2-D grid).
We have also implemented an alternative parallelization strategy using OpenMP threading that may be used for concurrent simulation of multiple grid cells within a node or process.As opposed to the full MPI parallelization, OpenMP avoids some of the overhead inherent in MPI (e.g., communication, data transfer).On some machines, the use of hyper-threading with OpenMP may result in better scaling than with MPI.Finally, the addition of OpenMP allows for hybrid OpenMP-MPI parallelism wherein OpenMP and MPI are used in concert.This advanced form of parallelism utilizes MPI for inter-node communication and OpenMP for shared-memory threading on-node.

Input and output
The image and CESM drivers use Unidata's NetCDF library for all I-O related to input parameters, meteorological forcings, and model output.NetCDF is a software library that reads and writes self-describing array-oriented scientific data with its metadata.It has been widely adopted among the geoscientific modeling community and provides a convenient self-describing machine-independent data format.Interfaces to the NetCDF library are available in many scripting languages including Python, R, Matlab, and Julia.Output files from the image and CESM drivers are formatted such that they meet the NetCDF Climate and Forecast (CF) metadata conventions version 1.6 (Eaton et al., 2009).
All drivers share a common logging module that greatly improves VIC's ability to self-document each model experiment.This logging module records the model version, input files, model settings, and performance statistics from key parts of the model source code.The verbosity of the logging module is controlled by a compile-time option.During model development, the logging module also facilitates debugging by providing full stack traces and line numbers when the model exits unexpectedly.When using MPI for parallelization, the logging module provides MPI task granularity when logging messages.
The classic driver continues to support the legacy ASCII and raw binary input and output file formats.We have also added additional metadata to the header of the ASCII file format but it has basically remained the same since VIC-4.
VIC previously included a meteorological forcing processor that made use of the MT-CLIM algorithms to estimate radiation and humidity from daily inputs of precipitation and temperature and which created sub-daily forcings from daily inputs.In VIC-5, this forcing processor has been removed from the model to facilitate a number of key enhancements that are described below.
1. Exact restarts.In the versions of VIC that included the MT-CLIM meteorological disaggregator, exact restarts were not possible due to a rolling time average in the MT-CLIM algorithm.By removing MT-CLIM in VIC-5, it is now possible to end a run, save the model state at that time step, and then start a new model run with exactly the same model state.This improvement has widespread applicability in hydrologic modeling and is particularly important for forecasting and climate modeling applications.
2. Identical treatment of forcings between drivers.The development of the image driver, which runs in a spacebefore-time loop order, was not compatible with MT-CLIM's loop order, which requires a time-before-space order.
3. Improved transparency in the forcing generation process.While VIC's core function is as a hydrologic model, specific modeling decisions were made when MT-CLIM was chosen as VIC's meteorological preprocessor and those decisions were abstracted deep within the model source code.As a result, the implications of forcing VIC with only minimum and maximum temperature and precipitation were abstracted away from and mostly invisible to the user.Removing the preprocessor from VIC enhances the transparency of the model by forcing users to make specific choices about the source of the model forcings.
A result of this change is that the input forcings must now have the same time step length as the model simulation.Consequently, for offline simulations based on widely used gridded daily observations (e.g., Livneh et al., 2015), forcing disaggregation must occur outside the model as a preprocessing step.A Python-based stand-alone disaggregation tool, MetSim (http://metsim.readthedocs.io,last access: 20 August 2018), has been developed for this purpose.
VIC-5 has also eliminated all hard-coded model parameters (e.g., emissivity of snow and the temperature lapse rate) from the body of the source code and collected physical constants into a single header file.Users may now specify an optional "parameters" configuration file to override default model constants that were previously hard-coded values.These changes follow a push in the hydrologic modeling community for hydrologic models to expose constant parameters (e.g., Mendoza et al., 2015) and greatly improves the transparency, flexibility, and accessibility of the model.

Testing
VIC-5 introduces a comprehensive test suite, now distributed with the model source code.The VIC test suite was designed to serve three primary purposes: (1) automated diagnostics and benchmarking of model performance, following Luo et al. (2012); (2) continuous integration providing automated build and unit tests to ensure reproducibility; and (3) the enabled development of contributions from a broader user community by facilitating consistency between model developers.As the VIC developer community continues to grow, this test suite will be an important tool in maintaining the function of the supported drivers.
The test suite is made up of seven components.The Build, Unit, System, and Examples tests are routinely run using the Travis CI continuous integration system (https: //about.travis-ci.com/,https://travis-ci.org/UW-Hydro/VIC, last access: 20 August 2018) in which the VIC test suite is executed using multiple compilers (GNU and Clang), MPI libraries (MPICH and Open-MPI), and computational architectures (Linux and OS X).VIC's Travis CI workflow includes the following steps: (1) selection of virtual machine with preinstalled environments, (2) download of VIC source code and test datasets from GitHub, (3) compilation of specific driver along with Python unit tests, and (4) execution of test simulations for the specific driver implementation (e.g., image).If any of the tests in these steps fail for any part of the test matrix, Travis CI returns a failed test status to the VIC GitHub page.This functionality is routinely used in evaluating VIC modifications that are suggested on GitHub via pull requests.VIC-5 has also been tested with the PGI, Intel, and Cray compilers; however, they are not used by default in the Travis CI test system.

Model distribution and documentation
Beginning in 2013, the VIC source code has been publicly available on GitHub (https://github.com/UW-Hydro/VIC, last access: 20 August 2018).GitHub is a web-hosting service built on top of the Git version control system and is meant to facilitate collaborative software development.VIC development uses many of the features available from GitHub including user and administrator controls, code review tools, and a bug and issue tracker.The ability to provide detailed, line-by-line code reviews has been particularly useful in the process of developing VIC-5 with multiple developers who have varying familiarity with the VIC source code.
We have adopted the widely used "GitFlow" branching model and workflow for managing the VIC Git repository.The workflow provides guidance for managing development branches and releases.Key features of the workflow are the separation of the master branch from development, feature, support, and bug-fix branches.This workflow is described in detail as part of VIC's online documentation.
www.geosci-model-dev.net/11/3481/2018/Geosci.Model Dev., 11, 3481-3496, 2018  for four sample locations for VIC-4 (red) and VIC-5 (blue) to FLUXNET observations (gray).The shaded range represents the interannual variability ( ± 0.5 SD).The r 2 statistic shown in the upper left corner is calculated using the hourly time series from the observations and model output from VIC-4 and VIC-5.The r 2 statistic comparing VIC-4 and VIC-5 is 1.00 for all sites and is not shown in this figure.
Finally, VIC has adopted the semantic versioning system (Preston-Werner, 2018) for determining new release names.In short, beginning with VIC version 5.0.0, version numbers follow a pattern of "MAJOR.MINOR.PATCH" in which MAJOR versions represent when changes are included that are not backward compatible, MINOR versions represent when features and new functionality are added in a backwards-compatible manner, and PATCH versions represent when backwards-compatible bug fixes are made.
As part of the development of VIC-5, we have implemented an updated, comprehensive documentation website for VIC (http://vic.readthedocs.io,last access: 20 August 2018).While previous versions of the VIC model did have a documentation website, edits were only possible by the VIC administrator, and the documentation was maintained separately from the model source code.In VIC-5, the documen-tation content is part of the VIC source code repository and is hosted on ReadTheDocs (https://readthedocs.org/, last access: 20 August 2018), a public documentation hosting service.A key benefit of the updated documentation is the ability to provide versioned documentation that matches specific model releases.This website is a particularly valuable tool for new users as it combines tutorials for how to set up and run VIC with a detailed description of the model itself.

VIC-5 simulations
The science, release, and performance tests described above were performed prior to the release of VIC-5.The tests were used to evaluate the final model version and were archived for future comparisons during the model development process.This section, along with Table 2, describes the specifics of the test simulations.

Science tests
The scientific verification of the VIC-5 release included point comparisons of VIC simulations with 66 AmeriFlux sites (Baldocchi et al., 1996(Baldocchi et al., , 2001;;Bohn and Vivoni, 2016) and 448 SNOTEL sites.These sites cover a range of hydroclimates across North America, providing a sufficiently large collection of environments for testing purposes.For the VIC-5 release, these datasets were used to ensure that the refactor did not significantly modify the model behavior relative to VIC-4.Future model development will use these archived test outcomes to motivate and evaluate scientific changes to the physical core of the model.

Release tests
We have designed a series of release tests that may be used to benchmark the VIC model during the development and release process.The release tests are continental-scale do-main simulations performed using typical model settings.
The specifics of the release test simulations for the VIC-5 release are described in Table 2.

Performance tests
We tested the parallel scaling performance of the image driver using a series of model simulations on the 50 km near an equal-area pan-Arctic domain used as the standard configuration in RASM.This model domain is shown in Fig. 4 and is detailed in  mode, in which the energy balance is not explicitly closed, resulting in a computationally less expensive model.Conversely, configuration B represents a much more complex model, in which the energy and water budgets are solved together using an iterative solver.Both configurations were run over the same model domain with the same meteorological forcings, with differences only in the model configuration settings as described above.Both model configurations were also tested using a hybrid OpenMP-MPI configuration with a combination of shared memory threading and parallel processes.Because memory requirements are modest for VIC, we do not include any memory profiling of the VIC parallelization schemes.

Results and discussion
The inclusion of a comprehensive science testing framework within the VIC-5 repository, as introduced in Sect.2.7, facilitates continuous and repeatable validation of the model throughout the development process.Figure 2 shows a summary graphic from the SNOTEL component of the science testing package.This figure demonstrates that VIC-4 and VIC-5 are producing basically identical (r 2 > 0.99) results for snow water equivalent (SWE) and that the two model versions have similar performance statistics when compared to observations.Figure 3 demonstrates an example comparison of latent and sensible heat from the FLUXNET component of the science testing package, showing a comparison between VIC-5 and flux tower observations for a subset of the sites included in the dataset.These automated diagnostics are particularly important throughout the model development and evaluation process to ensure that changes to the core physics of the model do not result in unintentional outcomes in other parts of the code.Additionally, these tests help provide baseline diagnostics for future VIC releases.
Improving the scaling performance of VIC-5 by adding parallel processing functionality enables VIC-5 to be used for more computationally expensive applications, such as highresolution modeling studies, sensitivity analyses, and coupled earth system models (e.g., RASM) that have shorter coupling intervals (e.g., 20 min) than the time steps that have traditionally been used in VIC (hourly to daily).Figure 5 demonstrates the scaling performance for the two model configurations.For the simpler model configuration (A), VIC run with MPI shows high throughput, but extending parallelization beyond four to eight nodes (144 to 288 cores) provides little speedup.This behavior is indicative of a program that is I-O limited.For the more complex model configuration (B), VIC has significantly lower throughput, but scales efficiently beyond 32 nodes (1152 cores).In this case, the physics code is much more time intensive than the I-O code of the model.For example, a comparison of the timing logs for two simulations that each use 144 MPI tasks shows that configuration A spends 48 % on I-O operations, while configuration B spends only 28 %.In practice, the scaling performance for most parallel VIC applications is predominantly a function of a computer's interconnect and disk speeds.We expect that VIC's scaling performance would therefore be improved on machines with more performant file systems (e.g., solid-state disks) or faster interconnects.
We also evaluated the scaling performance of the hybrid OpenMP-MPI parallelization in VIC and found that these simulations do not perform as well as the pure MPI implementation.Figure 5 also includes the hybrid OpenMP-MPI test simulations.One probable explanation for this is that the MPI libraries on the Thunder supercomputer are optimized such that the overhead of switching between MPI and OpenMP is larger than the performance improvements gained by shared memory threading.We expect that for other combinations of model configuration, machine, compiler, and MPI and OpenMP libraries, this hybrid parallelization scheme will perform better than pure MPI.
Additional performance enhancement in the scaling of the VIC image and CESM drivers will likely occur through the application of parallel I-O.Collective parallel reading and writing of NetCDF files, made possible through the NetCDF-4 and HDF5 libraries, is expected to dramatically reduce the amount of data passed via MPI processes.For computers with high-performance parallel disk systems, parallel I-O should allow the model to scale well beyond the limits shown in Fig. 5.The development of this feature has been reserved for later releases in the VIC-5 development track.

Conclusions
The VIC model continues to serve a broad user community.The VIC users email listserve (vic_users@u.washington.edu,last access: 20 August 2018) has more than 420 active members, and the analytics from the VIC website and documentation indicate that there are upwards of 1000 individual VIC users.The original VIC paper (Liang et al., 1994) continues to be widely cited in the hydrologic modeling literature with nearly 2300 citations as of January 2018.The VIC source code was moved to a public GitHub repository in 2013.Since then over 160 users and developers have checked out "forks".Additionally, we are aware of multiple ongoing developing efforts by the VIC community to implement new features.The open-source framework that we have implemented will facilitate mechanisms for the integration of model improvements from the greater VIC community into future VIC releases.
Beginning in 2014, the VIC model has been licensed with the open-source GNU General Public License version 2. The motivation behind making the model source code completely open source was to encourage participation by the model user and development community at large and to increase scientific transparency in the model development and application process (Ince et al., 2012).The VIC source code now uses the Git version control system (Torvalds and Hamano, 2010) and is publicly available on GitHub (https: //github.com/UW-Hydro/VIC,last access: 20 August 2018).While the majority of the VIC-5 development was done by researchers at the University of Washington, a number of features, additions, and bug fixes were made by unsolicited contributors.For example, the implementation of the RVIC streamflow routing extension within VIC (Sect.2.4) was a community contribution and provides a poignant illustration of the benefits of convenient open-source collaboration.
The development of VIC-5 focused on refactoring the legacy source code to support a range of modern modeling applications.While the release of VIC-5 does not modify existing or add new process representations, it does modernize the supporting infrastructure in the model while preserving legacy functionality.The incorporation of state-ofthe-art software engineering tools, such as GitHub and continuous integration, will facilitate the ongoing development and maintenance of the model by the broader VIC developer community.
We expect VIC-5's infrastructure improvements, combined with a suite of modern software tools, will help guide future VIC development and enable new applications of the model.For example, the enhancements related to highperformance computing will enable the application of the VIC model across larger, higher-resolution domains or for larger sensitivity experiments.Moreover, the improvements to the development process that were part of the path to VIC-5 represent a key component of the long-term sustainability of the model source code and the continued availability of the model as a tool for the modeling community.

Figure 1 .
Figure1.The reconfigured VIC-5 source code structure.The "physical core" includes the scientific modules of VIC (e.g., routines that simulate the fluxes and states of the model) and is used by each of the four drivers.The remainder of the VIC source code structure is considered driver-level code.Each driver uses components of the shared driver to minimize code duplication at the driver level.Likewise, the shared image driver includes source code that is used by both the image and CESM drivers (e.g., MPI and NetCDF utilities).Finally, the "extensions" constitute optional and driverspecific sub-models.

Figure 2 .
Figure 2. Example summary figure from the VIC-5 SNOTEL science tests.Panels (a) and (b) compare observed snow water equivalent (SWE) from SNOTEL sites to simulated SWE from VIC-4 (a) and VIC-5 (b).Panel (c) compares the results from the two VIC versions.

Figure 3 .
Figure 3. Example summary output from the VIC-5 science testing package comparing the annual cycle of latent heat (a) and sensible heat (b)for four sample locations for VIC-4 (red) and VIC-5 (blue) to FLUXNET observations (gray).The shaded range represents the interannual variability ( ± 0.5 SD).The r 2 statistic shown in the upper left corner is calculated using the hourly time series from the observations and model output from VIC-4 and VIC-5.The r 2 statistic comparing VIC-4 and VIC-5 is 1.00 for all sites and is not shown in this figure.

Figure 4 .
Figure 4.The 50 km near the equal-area Regional Arctic System Model (RASM) domain.The model domain is comprised of 25 996 grid cells.Shading denotes mean annual evapotranspiration from a test simulation run with model configuration (B) as described in Sect.3.

Figure 5 .
Figure 5. Model throughput for the two model configurations described in Sect. 3 run on the Thunder (SGI Ice X) supercomputer.The MPI-only simulations are shown in blue and the hybrid OpenMP-MPI simulations are shown in orange.

Table 1 .
Examples of VIC applications.
1. Build.The build tests ensure that VIC may be compiled and linked using a variety of different libraries and compilers.Given the increased complexity of the software stack supporting VIC, these tests are important to ensure portability of the model.Release.The release tests are longer, full domain (e.g., North America, global), image driver simulations performed prior to each model release.These tests demonstrate the current model behavior and are compared to previous releases to evaluate the changes in the model physics.These tests are archived for each model release to allow for retrospective model comparison.
2. Unit.The unit tests check function-level source code behavior.Using the same C-to-Python bindings used in the Python driver, many of the individual functions in VIC are tested using the "pytest" unit test framework (http: //doc.pytest.org,last access: 20 August 2018).These tests target individual functions in the physical core of the model and the shared driver.Specific tests for each driver are handled in the system tests.

Table 2 .
Summary of tests and datasets.

Table 3 .
Hardware used in VIC image driver parallel scaling performance tests.