New insights on the suitability of NetCDF/HDF5 as storage format for climate cloud repositories

Climate data analysis increasingly relies on cloud infrastructures to offer new and efficient methods of accessing the necessary climate data. Cloud repositories allow access to such data in a remote data access basis, which allows users to retrieve and manipulate their data without requiring file downloads, reducing storage costs resulting in more efficient systems. Together, climate cloud repositories and remote data access are evolving fast, due to the necessity of collaboration that brings together diverse communities to address challenges in climate science. In recent years, a prevailing discourse has emerged suggesting that traditional climate data storage formats are inherently unsuitable for remote data access. In this work, we present new insights that challenge this discourse and demonstrate that established storage formats such as NetCDF/HDF5 can continue to operate efficiently in cloud environments when accessed remotely. These findings contrast with the widespread perception that such formats are inherently unsuitable for cloud based workflows. In the context of the onset of CMIP7, these insights have the potential to substantially enhance climate data access and analysis for the broader research community without incurring major maintenance burdens.

1 Introduction

The predominant storage format in climate science has been NetCDF/HDF5, meaning that NetCDF serves as the library interface to files stored in HDF5 (Rew et al., 2006). We abbreviate NetCDF/HDF5 as NetCDF, since we do not consider earlier storage formats of NetCDF in the scope of this work. A NetCDF file may contain several multidimensional arrays, or variables, where each multidimensional array might be a field variable or a support variable (Hassell et al., 2017). Field variables include physical variables of the climate system, such as surface temperature, precipitation or wind speed. Support variables locate field variables in space and time. Field variables account for the majority of data to be stored, while support variables currently represent an almost negligible amount of data relative to field variables. Climate model output consists primarily of gridded data, meaning that it comprises field variables that represent physical variable values over cell boundaries and their associated support variables.

For several decades, NetCDF has been the predominant storage format for climate datasets, owing to its efficiency in storing dense climate data (Cimadevilla, 2025). Climate data produced within the framework of Model Intercomparison Projects (MIP) are generated by global or regional climate models. The Earth System Grid Federation (ESGF) has arisen as the federated solution to share climate datasets worldwide, due to the large amount of data available (Cinquini et al., 2012). The emergence of cloud infrastructures, in contrast to traditional High Performance Computing (HPC) environments that form part of the ESGF, has raised questions regarding the suitability of NetCDF as a storage format for cloud based workflows based on remote data access (Abernathey et al., 2021). The prevailing discourse has suggested that NetCDF is not well suited to the cloud and that alternative formats and workarounds, such as Zarr and Kerchunk, are necessary if climate data are to be effectively stored and accessed in a remote data access basis, either from the ESGF or from cloud repositories.

In this work, we address the factors that have been overlooked and have contributed to the perception that NetCDF is unsuitable as a storage format for remote data access and climate cloud repositories. We demonstrate that the inefficiencies in common NetCDF usage arise from multiple factors, including non-optimized chunking and the organization of metadata structures used to index these chunks, together with the lack of client libraries to take advantage of internal data structures. We provide detailed, low-level explanations of how common usage interacted with low-level structures to result in observations of poor remote access performance to NetCDF data. Furthermore, we develop and describe the tools and practices necessary to enable the performant use of NetCDF in cloud environments. Finally, we evaluate and analyze the performance of different technologies used to achieve efficient remote data access.

2 Background

Climate data analysis has traditionally involved downloading NetCDF files to local workstations or computing infrastructures for subsequent analysis. This approach relies on certain simplifications that must be reconsidered in other modes of data access, as in the context of remote data access. Data access to locally downloaded NetCDF files offers the advantages of relative simplicity and high performance, since latency to randomly access a NetCDF files is limited to low-latency data transfer within a local workstation or local area network (LAN), in contrast to high-latency remote data access that depends on wide area networks (WAN). However, the file downloads method presents significant challenges for analysts, as it requires substantial time and effort to manage data acquisition and download. Tasks such as downloading and organizing large volumes of files often divert attention and resources away from core research activities.

Climate cloud repositories leverage cloud optimized storage formats and remote data access mechanisms to enable researchers to query and analyze data directly on remote servers or cloud platforms without transferring entire datasets (Abernathey et al., 2021). This capability, commonly referred to as subsetting, allows specific workflows to retrieve only a limited portion of a dataset, thereby reducing the volume of data transmitted over the network relative to a file download approach. Consequently, spatial subsets of large domains or selected temporal segments of extended time series can be analyzed without downloading entire files that encompass the full spatiotemporal extent of the data. However, achieving such efficiency requires selecting storage format layouts that support performant remote access and data subsetting. The primary technique for achieving this is chunking, which relies on internal data structures to index the chunks stored within the file. The organization and storage of these indexing structures can, in turn, have a significant impact on performance if not managed properly.

Libraries that store multidimensional arrays make use of chunking to allow compression of data and more efficient data access and storage. Because storage media are inherently single dimensional, whereas climate data are multidimensional in nature, multidimensional arrays that contain climate data must be transformed from their multidimensional spatiotemporal representation into a single dimensional storage space. This transformation entails a critical trade off, as the relationship between data access patterns and storage layout directly determines data retrieval latency. The resulting performance differences can span several orders of magnitude.

However, efficient chunk usage requires not only good selection of chunk shapes by the user, but optimal use of the chunk indexes within the storage layout. Chunk indexing can be implemented in several ways, depending on the storage model adopted by a given format or library. Formats such as Zarr employ an indexing scheme that requires the shape of the multidimensional array and its chunk shape as input, allowing the location of a chunk to be computed directly from these two parameters. In Zarr, the location of a chunk is not represented as a byte position within a linear file, but rather as a key in a key-value storage system. This key is derived from the geometric chunk's position within the multidimensional array, with the individual coordinates typically concatenated using a delimiter. Figure 1 provides a conceptual visualization of Zarr chunk indexing mechanism.

Figure 1 Chunk indexing based on chunk coordinates. Each cube is labeled with the numerical index of the corresponding chunk, with each cube representing a chunk of arbitrary size. On the right, the coordinates of each chunk within the three-dimensional space of the multidimensional array are shown. In formats like Zarr, chunk coordinates are usually applied to storage unit names, files or objects, to allow easily identification of chunks.

Formats such as HDF5 use data structures contained within the file itself to index chunks within the file's linear address space. This is required because the data might be written incrementally in no specific order and it is not desirable to reserve empty space in advance. In this indexing mechanism, the chunk index nodes contain several entries that point to byte locations where chunks are located within the file. NetCDF/HDF5 relies on B-trees to provide a flexible storage solution for objects that grow in a way that leads to discontiguous storage of chunks. B-trees are self-balancing search trees that maintain sorted data and allow searches, sequential access, insertions, and deletions in logarithmic time (Bayer and McCreight, 1972). They are widely used in databases and file systems due to their efficiency in handling large amounts of data (Comer, 1979). Figure 2 provides a conceptual visualization of chunking and chunk indexing based on a B-tree index.

Figure 2 Illustration of chunk indexing based on a B-tree. A chunk index is a data structure that maps each chunk identifier to its position in linear storage. In the context of this work, a chunk index is implemented by a B-tree, whose nodes contain entries that may be scattered throughout the linear space of the NetCDF file. Arrows in the B-tree represent either pointers to other nodes or pointers to raw chunks, both located arbitrarily in the space of the file.

Chunk indexing is as important as chunk shape in terms of its performance implications. The choice of chunk indexing mechanism can affect performance in several ways. For example, indexing chunks using the naive Zarr approach results in substantial inode consumption in the storage system. This is because each chunk is typically stored as an independent unit of storage, corresponding to a file in a file system or an object in an object storage system. In the case of Zarr, the trade-off is as follows: chunks are independent of one another, and no locking mechanism is required, which simplifies development for certain applications. However, this approach can result in substantial inode consumption in file-system-based storage systems, making Zarr highly inefficient under such conditions. For these cases, in version 3 Zarr introduced optional sharding, which packages multiple chunks into a single storage unit, thereby reducing inode consumption.

Chunk indexing based on B-trees usually follows a mechanism in which the B-tree data structure responsible for indexing chunks is stored within the same storage unit used to store the chunks. In the case of NetCDF/HDF5, this storage unit is an HDF5 or NetCDF file. This approach avoids the problem of high inode consumption because the file system is not aware of the internal storage of the chunk index, which is instead managed by the client library. However, the data structure may be stored in such a way that its components are dispersed throughout the linear address space of the file. As a result, retrieving the chunk index can be expensive in terms of the number of requests required to retrieve the complete index from the storage system.

To understand why NetCDF has come to be regarded as unsuitable for storing climate data in the cloud for remote access, it is essential to examine the internal data structures of a NetCDF file, particularly those used to index chunk locations. We provide evidence that, by organizing these data structures appropriately, NetCDF can be considered cloud-optimized and effectively employed as a storage format for climate data in cloud repositories.

3 Methods

This work is based on the hypothesis that it is possible to achieve performance comparable to that of other cloud-optimized formats when using NetCDF. It further examines the factors that must be considered in order to make NetCDF performant in cloud environments. We selected a small set of NetCDF files produced in the context of the Coupled Model Intercomparison Project (CMIP6) that represent different file-size scales and temporal resolutions (Eyring et al., 2016). Our primary example uses the uas variable, which represents the u-component of the near-surface wind. We arbitrarily selected this 2D variable primarily to simplify the analysis and presentation of the results.

Although 3D variables, such as ua, account for a substantial fraction of atmospheric NetCDF data, the dimensionality and physical meaning of a variable do not fundamentally change the mechanism used to index its chunks. For a given chunking scheme, the B-tree structure is determined primarily by the number of chunks that must be indexed. Consequently, the same B-tree behavior applies for 2D, 3D, and higher-dimensional variables. We conducted some testing on 4D variables but we chose not to include results since they are redundant with the files used with different time scales. We account for differences in the number of chunks and file sizes by considering examplar files with different temporal resolutions. Our selection of datasets with different temporal resolutions provides a basis for generalizing and extrapolating how variations in metadata location affect the internal data structures of NetCDF files and, consequently, their performance. Three files were chosen, with monthly, daily, and three hourly temporal resolutions. Table 1 summarizes the original NetCDF files, as downloaded from the ESGF, that are considered in this study.

Table 1 List of the original NetCDF files as obtained from the ESGF. The selected files sample a range of temporal resolutions in order to account for differences in the size of the internal data structures of HDF5. It can be observed from the metadata column that the size of the internal data structures increases rapidly as a greater number of chunks must be indexed for the NetCDF variables. Note that the field variable and time coordinate of these files are chunked with one spatial latitude/longitude chunk per time step. The chunks column refers to the number of chunks of the field variable only (uas).

These original files are generated within HPC infrastructures and are intended for use in a low-latency file downloads approach rather than a high-latency remote data access setting. As a result, their internal data structures and B-trees are positioned arbitrarily within the NetCDF file. This arrangement does not pose a problem when analyzing data on a local workstation, where the latency associated with accessing B-tree fragments is negligible. However, in a remote data access context, the latency cost of retrieving many small byte segments from disparate locations becomes substantial. This factor has been largely overlooked in the literature when accounting for the high cost of accessing NetCDF files from cloud repositories. Figure 3 illustrates the locations of B-tree fragments in the original NetCDF files as downloaded from the ESGF.

Figure 3 Stacked histogram of the locations of all B-tree fragments in the original CMIP6 files, given by B-tree offsets in bytes from the start of the file, showing that for the uas and time, and time_bounds variables they are scattered throughout the whole NetCDF files. The x-axis shows that B-tree fragments occupy the full space of each original NetCDF file.

Table 1 highlights several factors that contribute to the inefficiencies associated with accessing these NetCDF files when they are uploaded directly to the cloud and accessed through remote data access. First, the daily and three hourly files contain a substantial volume of metadata, which in this context refers to internal HDF5 structures and not user attributes. Although this metadata is relatively small compared with the total size of the NetCDF file, it is sufficiently large to generate inefficiencies in remote access scenarios. The daily file contains approximately 25 MB of metadata, which increases to more than 40 MB in the three hourly file. However, the principal source of inefficiency is not merely the size of the metadata, which could be retrieved within a few seconds if it were stored contiguously. Rather, the key factor is that these metadata bytes are scattered throughout the whole linear space of the NetCDF file, as displayed in Fig. 3. Consequently, retrieving the metadata requires thousands of individual requests, which, in high-latency environments such as remote data access, renders data access effectively impractical. This issue also affects smaller files, such as the monthly NetCDF file from ESGF shown in Table 1. Although it contains a relatively small amount of metadata, the fact that these metadata are scattered throughout the file makes retrieving subsets practically intractable.

If these metadata bytes were stored at predictable locations within the linear storage space of the NetCDF file, clients could exploit this organization to retrieve all metadata through a single read operation. This would avoid the inefficiencies that currently make such files impractical to analyze when stored in the cloud. We refer to files whose metadata bytes are scattered throughout the file as fragmented, in analogy to traditional file system fragmentation that occurs in conventional computer systems. We refer to the process of modifying NetCDF files with the aim of reallocating these bytes to appropriate locations as repacking.

Files that have undergone this procedure are described as having consolidated metadata or being defragmented, meaning that their internal data structures are placed at the beginning of the file, where clients can retrieve this information in a single read operation. However, it is important to notice that climate data producers may produce NetCDF with consolidated metadata without the need for repacking, given that they configure their tools properly. At the moment the only tools known to the authors that do it by default are cfdm and cf-python (Hassell et al., 2017). We will see that to fully leverage modern WAN networks and maximize bandwidth utilization, clients must not only understand the internal data structure of a NetCDF file for efficient remote data access, but also implement appropriate concurrency mechanisms to retrieve data chunks in parallel.

4 Validation

Under the current assumptions, the available mechanisms for efficient remote access to climate data involve either transforming the data into Zarr format or generating a Kerchunk file for the NetCDF dataset, which effectively means creating an extended copy of the NetCDF B-tree index. Kerchunk provides a Zarr interface to existing NetCDF files, enabling efficient data access without duplicating the full dataset, as it stores only the locations and sizes of the chunks within the NetCDF files. As a result, a Kerchunk file represents only a small fraction of the original NetCDF dataset, although, as discussed below, this size is not negligible under current chunking practices within the climate data community. In this sense, Kerchunk consolidates metadata without requiring reorganization of the original files, albeit at the expense of managing the resulting Kerchunk files. This cost might be far from insignificant, as we shall see. Table 2 shows the corresponding Kerchunk files for the original NetCDF files obtained from the ESGF.

Table 2 File size and number of references contained within the corresponding Kerchunk files generated from NetCDF datasets obtained from the ESGF. The large size and number of references result from historical suboptimal chunking practices within the climate data community. Alternative Kerchunk formats, such as Parquet, aim to reduce the size of the resulting files but do not address the underlying issue, namely the presence of an excessively large and unnecessary number of references.

We will provide evidence that properly repacked NetCDF files, when accessed with suitably optimized clients, can support efficient remote data access without the need for Kerchunk files. Moreover, we illustrate how repacking favors smaller and more manageable Kerchunk representations by reducing their file size while improving access performance, thereby enhancing their usefulness when generating aggregations from multiple NetCDF source datasets. Thus, other uses of Kerchunk such as generating virtual aggregations of multiple NetCDF files would greatly benefit from repacked NetCDF files.

4.1 Repacking

We performed repacking of the original files obtained from the ESGF using cmip7repack (Hassell and Cimadevilla, 2025). This tool is designed to organize the internal data structures of an HDF5 file so that they are placed at known and predictable locations at the beginning of a NetCDF file. After repacking, the metadata is both reduced in size and relocated to the beginning of the file. These changes have distinct effects on performance. The cmip7repack utility is essentially a wrapper around the HDF5 h5repack utility, which performs the actual repacking, including modifying the metadata block size and the chunking scheme of individual variables. However, h5repack can also be used independently to tailor these settings to the specific needs of individual datasets, thereby extending the applicability of the results of this work to any data stored in HDF5 format.

The relocation and improved locality of the metadata, including the B-tree chunk-index structures, primarily reduce the latency associated with retrieving the chunk index, which must be accessed before the corresponding data can be read. In addition, using larger chunks, together with rechunking coordinates into a single chunk, substantially reduce the total number of chunks and, consequently, the number of entries and the size and depth of the B-tree structures used for chunk indexing. This reduction in the number of chunks also decreases the number of remote requests required to retrieve the data, thereby improving data-transfer efficiency and throughput. Thus, the performance improvement resulting from repacking reflects both the reduced number of larger chunks and the resulting simplification of the B-tree structures, as well as the more contiguous placement of the remaining metadata near the beginning of the file. Table 3 illustrates the benefits of repacking the original NetCDF files while specifying a chunk size of 4 MB (before compression) for the field variable (uas).

Table 3 List of the repacked NetCDF files obtained after applying cmip7repack to the original files. Repacking was performed by specifying chunk sizes of 4 MB for the climate variable (uas). It can be observed that the size of the internal HDF5 data structures decreases significantly as the chunk size increases.

It is also important to note that support variables that were originally stored with suboptimal chunking are now stored in a single larger chunk, thereby eliminating the associated B-tree fragments previously required to index the chunks of those variables (time and time_bounds). The most striking example of the impact of rechunking support variables into a single chunk can be observed in the three hourly file, which originally required 292 192 chunks for each of the uas, time, and time_bounds variables. This configuration accounted for the majority of the 40 MB of metadata stored in the file. This also has the effect of substantially reducing the size of the generated Kerchunk files for these NetCDF datasets, which may still be desirable for constructing aggregated datasets from multiple NetCDF files. Table 4 shows the information for Kerchunk files of repacked datasets.

Table 4 File size and number of references contained within the corresponding Kerchunk files generated from NetCDF datasets that have repacked with cmip7repack. By using larger chunks and storing support variables within a single chunk, both the file sizes and the number of references are substantially reduced in comparison with the original Kerchunk files.

By storing the support variables in a single chunk and using a chunk of size $(\mathrm{50},\mathrm{143},\mathrm{144})$ instead of the original $(\mathrm{1},\mathrm{143},\mathrm{144})$ for the uas variable, the number of B-tree nodes and entries is reduced substantially. Consequently, for the 3-hourly file the volume of metadata decreases from the original 40 MB to approximately 355 KB. More importantly, this metadata is now located at the beginning of the file rather than being scattered throughout the entire file. Figure 4 illustrates the distribution of B-tree fragments when 4 MB chunks are used.

Figure 4 Locations of all of the B-tree fragments in the repacked CMIP6 files using 4 MB chunks for the climate variable (uas). Locations are given by B-tree offsets in bytes from the start of the file. In contrast to the original ESGF files, coordinate and support variables are stored in a single chunk after repacking, and all B-tree data is stored at the beginning of the file.

4.2 Performance

To assess the impact of repacking, we conducted a performance experiment aimed at evaluating the implications of properly formatted NetCDF files when accessed remotely over a WAN network, where a cloud repository located in the UK stores the data and these are accessed from a virtual machine client located in a private cloud provider in Spain. We aim to demonstrate that defragmented NetCDF files meet the criteria to be cloud efficient without the need for Kerchunk sidecar files. Despite the importance of repacking, it is important to note that repacking is a necessary but not sufficient condition to guarantee optimal performance in a remote data access context. To achieve full performance, software clients must satisfy at least two conditions: they must be able to exploit the structure of repacked NetCDF files, and they must be capable of reading chunks concurrently.

The first condition can be satisfied by instructing clients to request an appropriate amount of data from the beginning of the file, where the consolidated metadata are located. The second condition is more challenging to achieve. In this case, clients must employ parallelism techniques, such as asynchronous execution, multithreading, and multiprocessing, to enable parallel reading of chunks so that the available bandwidth can be effectively utilized. Common clients for accessing NetCDF data, such as netCDF4-python or h5py, do not support native parallel access to data. Thus, for the evaluation of performance of repacked NetCDF files we will utilize both Kerchunk/Zarr and Pyfive (Lawrence et al., 2026), a recent pure Python HDF5 reader developed to enhance access to climate data.

We designed the performance experiment in two parts. First, we demonstrate that the lack of repacking accounts for the poor performance often attributed to NetCDF in cloud environments. Second, we show that once a file has an appropriate structure – that is, when the B-trees are located at well-defined positions and a sensible chunk structure is used – overall performance depends on the ability of clients to implement concurrent chunk retrieval to saturate the available bandwidth. We compare the results obtained from both opening and loading the data using different configurations and clients. The different configurations are the following:

• Kerchunk (original async) – Usage of Kerchunk generated from the original NetCDF file as stored in the ESGF. We label it async because Kerchunk natively supports loading chunks in parallel using asynchronous HTTP requests.

• Kerchunk (repacked async) – Usage of Kerchunk generated from the repacked NetCDF file as stored in the ESGF. Again, we label it async because Kerchunk natively supports loading chunks in parallel using asynchronous HTTP requests.

• Pyfive (repacked serial) – Usage of the Pyfive library to load a repacked NetCDF file without parallelism when retrieving chunks.

• Pyfive (repacked async) – Usage of the Pyfive library to load a repacked NetCDF file using parallelism both for retrieving chunks and for applying the chunk filter pipeline.

• Zarr (original async) – Zarr store created from the original NetCDF file as stored in the ESGF. It uses Zarr without sharding, so each chunk is stored in a single storage unit (either an object or a file).

• Zarr (shards async, single shard) – Zarr store created from the repacked NetCDF file in which each variable is stored in a single shard whose shape matches that of the variable, so that all chunks fit into a single shard, thereby closely resembling a NetCDF file. This configuration is used only for experimental illustration and not for operational use.

• Zarr (shards async, shards of shape 1000) – Zarr store created from the repacked NetCDF file in which variables with a time dimension are divided into shards with a size of 1000 along the time dimension. This configuration is intended to illustrate the operational use of Zarr with sharding.

• Zarr (repacked async) – Zarr store created from the repacked NetCDF file without sharding, so that each chunk is stored in a single storage unit (either an object or a file).

Figure 5 presents the time required to open the different files and stores under different formats and configurations. The first observation in Fig. 5 is the absence of results for opening an original (non-repacked) NetCDF file using Pyfive. In this context, we define opening a file as the process of reading its basic metadata, including the B-trees of the chunked variables. Because the B-tree fragments in the original files are dispersed throughout the entire file space, as shown in Fig. 3, attempting to open such files becomes impractical. This limitation has contributed to the perception that NetCDF is inherently unsuitable for cloud storage. However, we report the time required to open the original NetCDF files using Kerchunk.

Figure 5 Measured mean time required to open different NetCDF files from an object store using various clients and configurations. Original NetCDF files are accessed only via Kerchunk, which compensates for the fact that the B-trees in the original files are dispersed throughout the file space, albeit at the cost of significantly larger Kerchunk files. Concurrent access to chunks is marked as async. Error bars display minimum and maximum times available in the sample.

Figure 5 illustrates the usage of Kerchunk, which is to provide a duplicated cache of the B-trees for the chunked variables. However, the time required to open the file is considerably higher than for repacked files in relative terms. This behavior is driven by the substantial size of Kerchunk files generated for the original NetCDF data and the large number of references they contain, as shown in Table 2. For all results, it should be noted that the observed variability is within the expected range when establishing remote connections to servers over a wide-area network.

The benefits of NetCDF files with consolidated metadata from the repacking process are clearly shown. Because the B-trees are located at the beginning of the NetCDF files, all clients that support the basic functionality of loading the initial bytes of a remote file can take advantage of this, enabling files to be opened within a few seconds. Thus, opening remote NetCDF files is transformed from a prohibitively time-consuming and potentially intractable task into a trivial one. Additionally, due to the rechunking of the original files into larger chunks, Kerchunk files become significantly smaller, allowing them to be loaded much more quickly. Thus, contrary to common belief, NetCDF can be natively performant in the cloud, provided that the internal structures of HDF5 are properly understood and configured to meet the requirements of remote data access over WAN networks. However, the performance of NetCDF cannot be assessed solely by the time required to open a file; it is also necessary to consider the latency and throughput when loading data from its variables. Figure 6 shows the time required to load the uas variable under different configurations and across various clients.

Figure 6 Measured mean time required to load the uas variable from different NetCDF files stored in an object store using various clients and configurations. The variable is accessed in its entirety, without a specific data access pattern, to illustrate the throughput capabilities of each configuration and client. Concurrent access to chunks is marked as async. Error bars display minimum and maximum times available in the sample.

The results highlight two key factors, beyond the previously discussed issue of fragmented B-trees, that have led the climate community to assume that NetCDF is unsuitable for cloud storage. First, the small chunk sizes defined in the original NetCDF files impose a major limitation on throughput. Chunks on the order of kilobytes incur significant latency costs, whereas larger chunks on the order of megabytes better utilize the network and can saturate the available bandwidth. Second, the lack of clients with the capability to fetch chunks concurrently represents another major limitation, as it prevents full utilization of the available bandwidth.

Figure 6 illustrates these limitations by using Kerchunk to access the original NetCDF files as published in the ESGF and Pyfive in serial mode, i.e., without concurrency mechanisms. Kerchunk provides an external representation of the chunk index by describing the locations of chunks in the original files, and therefore does not modify the native HDF5 chunk-index structure. In contrast, our repacking approach reorganizes the native HDF5 chunk index as part of the repacking process, with the resulting B-tree remaining an integral part of the HDF5 file structure. While Kerchunk can exploit concurrency through asynchronous HTTP requests, the small chunk sizes still limit the achievable throughput. Thus, Kerchunk can provide an alternative means of accessing existing files without modifying their internal structure, but it does not by itself address the performance limitations associated with poorly chosen chunk sizes or the organization of the native HDF5 chunk index.

To further assess the performance of formats other than NetCDF, we converted the NetCDF files into corresponding Zarr stores that matched the chunking configuration of the original datasets (i.e., identical chunk shapes, compression settings, and filter configurations). In evaluating Zarr, we considered stores that: (a) matched the configuration of the original NetCDF files, (b) matched the configuration of the repacked NetCDF files, and (c) employed different sharding configurations to account for performance differences between full HTTP requests and partial HTTP range requests. We observe that our storage system of choice delivers higher performance when Zarr stores chunks independently, whereas performance is reduced when sharding is used, as this requires HTTP range requests to access individual chunks.

The results demonstrate that remote data-access performance is not determined by the file format or library alone, but rather by the interaction between file organization, storage infrastructure, and the capabilities of the client used to access the data. This is illustrated by using Pyfive to access repacked NetCDF files without concurrency support, which does not achieve the performance obtained when concurrent requests are enabled. Similarly, the comparison with a single-shard Zarr store demonstrates that the underlying storage infrastructure can exhibit different performance characteristics when accessing independent objects compared with performing byte-range requests on a single object.

When appropriate chunking, metadata organization, and effective concurrency mechanisms are combined, repacked NetCDF files can achieve performance comparable to alternative cloud-oriented approaches. This is illustrated in Fig. 6, where Kerchunk is used to access repacked NetCDF files and Pyfive uses the concurrency support that we recently added and that is now enabled by default for remote access. When Kerchunk accesses a properly repacked NetCDF file, it can take advantage of the larger chunks, thereby improving bandwidth utilization and performance compared with accessing the original NetCDF files. In the case of Pyfive, the use of concurrency mechanisms similarly enables high performance, which was not achieved with a serial, non-concurrent approach, even when the NetCDF files were properly repacked.

Figure 7 Measured time required to load the uas variable from different NetCDF files stored in an object store using various clients and configurations. The variable is accessed in its entirety, without a specific data access pattern, to illustrate the throughput capabilities of each configuration and client. Error bars display minimum and maximum times available in the sample.

Figure 7 shows that the performance of Kerchunk based on repacked NetCDF files is significantly worse than that of Pyfive or Zarr, even though these Kerchunk references point to properly repacked NetCDF files rather than the original files. This makes Kerchunk far from optimal for large NetCDF datasets. Consequently, the common assumption within the climate community that Kerchunk can fully offset the limitations of fragmented NetCDF files should be treated with caution.

We also present the performance of Pyfive, which accesses repacked NetCDF files directly, and compare it against native Zarr stores. The results indicate that our storage infrastructure penalizes HTTP range requests, thereby reducing the performance of both Pyfive and sharded Zarr configurations. For this particular storage system, unsharded Zarr, in which each chunk is stored as an individual object, achieves the best performance, albeit at a considerable cost in terms of object/file maintenance once every chunk is a file/object (and detrimental performance on parallel file systems). Overall, these results illustrate that native NetCDF can in fact provide strong performance when properly configured and accessed, although achieving this required the development of a new library, Pyfive, capable of exploiting these features effectively.

5 Conclusions

Climate data access is undergoing rapid transformation in response to emerging requirements driven by increasingly collaborative research environments and the challenges associated with Big Data. Traditionally, climate data distribution has relied on the generation of NetCDF files at HPC centers where climate models are executed. Subsequent data analysis has typically been conducted either at these same centers or at external institutions that obtain the data by downloading NetCDF files to their local infrastructures. Within this framework, the ESGF has served as the primary federated infrastructure for the distribution of such datasets. More recently, cloud-based climate data repositories have emerged and demonstrated their effectiveness in enabling remote access to large-scale climate datasets. These platforms facilitate more efficient data sharing and significantly enhance collaboration among climate researchers and research initiatives.

Throughout these developments, NetCDF has increasingly been perceived as an inefficient format for remote data access. Consequently, prevailing narratives have favored alternative approaches, such as emerging formats like Zarr, or workarounds including Kerchunk sidecar files accompanying NetCDF datasets, as necessary to achieve acceptable performance in cloud-based repositories and remote access workflows. While these concerns are valid, the role of NetCDF’s internal data structures in contributing to these performance limitations has been largely overlooked in the literature. In this work, we have addressed this gap by providing a detailed examination of the internal organization of NetCDF files and identifying the structural factors that cause performance issues. Furthermore, we have introduced tools designed to mitigate these limitations and have demonstrated that NetCDF can achieve competitive performance in both cloud-based environments and remote data access scenarios.

With these new insights in mind, we believe that future operational decisions will no longer depend on the specific storage format (i.e., NetCDF or Zarr), as both can achieve comparable performance when properly configured. Instead, future operational management will focus on identifying and applying the appropriate configuration. Our results show that performance is strongly influenced by the number of underlying objects within a dataset, leading to a trade-off between the number of NetCDF files or Zarr objects/shards and the characteristics of the storage system. Object storage systems appear to benefit from a larger number of individual objects, whereas parallel file systems tend to favor configurations that enable efficient random byte-range reads. This trade-off in the number of objects – whether files, shards, or chunks – is also a critical factor for the performance of aggregation technologies such as Kerchunk. Consequently, object granularity emerges as an additional consideration alongside the more traditional trade-off associated with chunk shape and size.

Current developments within CMIP and the ESGF increasingly promote the use of Kerchunk to facilitate cloud-based access to NetCDF datasets. However, the findings of this work demonstrate that the mere generation of a Kerchunk file for fragmented NetCDF datasets is not sufficient to ensure efficient remote data access. Instead, performance depends critically on the internal structure of the NetCDF file, including factors such as the organization of B-tree nodes and the chosen chunking strategy. Only when these elements are appropriately configured can Kerchunk deliver proper efficient data access. At the same time, the maintenance of Kerchunk files introduces an additional layer of complexity within the data federation. For this reason, we advocate for a more sustainable approach based on tools that allow repacking or directly generate metadata consolidated NetCDF files, which organize the internal structure of NetCDF files to enable efficient remote access without the need for supplementary artifacts. The CMIP7 Model Output Specifications already require data producers to repack their datasets prior to data publication in the ESGF (Taylor et al., 2026). This strategy confines maintenance overhead to the one-time process of restructuring files prior to publication. Collectively, these findings provide a basis for more informed decision-making within the climate research community regarding data storage practices, with the ultimate goal of enhancing accessibility and usability for end users.