We have made available the script to generate the CSV files in the repository (tree.py, https://github.com/zequihg50/new-insights-netcdf/). Release v1.1.2 of Pyfive is required to test parallel access to data (https://github.com/NCAS-CMS/pyfive/releases/tag/v1.1.2). Per the HDF5 specification [1], the fields are defined as follows:

level - The node level indicates the level at which this node appears in the tree (leaf nodes are at level zero). Not only does the level indicate whether child pointers point to sub-trees or to data, but it can also be used to help file consistency checking utilities reconstruct damaged trees.

entry - Each entry (key and address of a child) among the entries used in the B-tree.

offset - Byte offset of each entry in the B-tree.

Fig-1: The purpose of B-trees is to grow over time as chunks are written. Because clients typically write one chunk at a time, space for future chunks is reserved in the index after the first chunks have been written. During repacking we increase the metadata block size and HDF5 attempts to place B-trees at the beginning of the file. We will clarify this in the reviewed version. For more information, please see https://github.com/Unidata/netcdf-c/issues/3386.

Table 1: We provide links to both the original and repacked files. Please note that repacking is not a deterministic operation, as its output may vary depending on the version and behavior of h5repack.

- https://thredds.climate.ifca.es/thredds/fileServer/tests/uas_Amon_IPSL-CM6A-LR_piControl_r1i1p1f1_gr_185001-234912.nc

- https://thredds.climate.ifca.es/thredds/fileServer/tests/uas_Amon_IPSL-CM6A-LR_piControl_r1i1p1f1_gr_185001-234912.nc_cmip7repack4mb

- https://thredds.climate.ifca.es/thredds/fileServer/tests/uas_day_IPSL-CM6A-LR_piControl_r1i1p1f1_gr_18500101-23491231.nc

- https://thredds.climate.ifca.es/thredds/fileServer/tests/uas_day_IPSL-CM6A-LR_piControl_r1i1p1f1_gr_18500101-23491231.nc_cmip7repack4mb

- https://thredds.climate.ifca.es/thredds/fileServer/tests/uas_3hr_IPSL-CM6A-LR_piControl_r1i1p1f1_gr_187001010300-197001010000.nc

- https://thredds.climate.ifca.es/thredds/fileServer/tests/uas_3hr_IPSL-CM6A-LR_piControl_r1i1p1f1_gr_187001010300-197001010000.nc_cmip7repack4mb

[1] - https://support.hdfgroup.org/documentation/hdf5/latest/_f_m_t4.html#subsubsec_fmt4_infra_btrees_v1

We thank the reviewer for the careful reading of our manuscript and for the positive and constructive summary of our work.

General comments

In the study, we effectively use only three example files to demonstrate our approach. It should be noted, however, that these files were selected to represent different temporal resolutions. Given the chunking scheme commonly used for CMIP data, where each chunk corresponds to one time step and contains all spatial locations, the temporal resolution represents the main factor affecting the size and distribution of the B-trees.

Regarding B-tree behaviour, 2D, 3D, and nD variables, as well as different climate variables such as temperature and precipitation, are expected to behave in the same way. The B-trees are affected primarily by the number of chunks that they need to index, rather than by the dimensionality or physical meaning of the variables. Consequently, the number of chunks, and therefore the B-tree structure, varies in our test cases mainly as a function of the temporal resolution, which we have already considered.

For this reason, we believe that adding further variables or netCDF files with the same chunking characteristics would largely replicate the results already presented, without providing substantially different insights. Nevertheless, we appreciate the reviewer’s suggestion and have clarified this point in the manuscript to better explain the rationale behind the selection of the test files and variables.

Specific Comments

Introduction: We agree that we have not paid sufficient attention to the issue of inode consumption associated with some data formats, particularly Zarr, in the manuscript. We will address this aspect in the revised version. High inode consumption is a well-known concern when using Zarr in HPC environments, particularly when data is stored as a large number of individual objects or files. This is one of the motivations behind sharded Zarr storage, which reduces the number of individual files. While we do not have empirical evidence to quantify the extent to which this issue affects cloud object storage, where the concept of inodes does not generally apply in the same way as in traditional file systems, the number of objects can nevertheless have implications for storage management, metadata operations, and access performance. We appreciate the reviewer’s suggestion and will discuss this aspect in the revised manuscript, including the advantages of HDF5 in avoiding the creation of additional files and, consequently, additional inode consumption in file-system-based environments.

l72: Please name here explicitly cloud-native storage formats you had in mind when writing.

We were referring specifically to Zarr and to other approaches that provide cloud-optimized access to netCDF data, such as Kerchunk. We will make this explicit in the revised manuscript.

Table1: What about rounding up to 26 MB? Or at least using a decimal point instead of a comma.

We will change it to 26 MB.

l84/85: How did you extract the B-tree fragments from the netCDF file?

We added the script “tree.py” to the GitHub repository, which extracts the locations of B-tree nodes and entries given the location of the root B-tree node. We obtained B-tree root node locations by debugging Pyfive while opening the selected files. The code of interest is located in https://github.com/NCAS-CMS/pyfive/blob/667c4681685ac32d5528c93a4058c733914dadb4/pyfive/btree.py#L133.

Figure 2: The tick labels and axis labels are too small. Please increase to make them readable at 100 % zoom level. I cannot see the colours for lat and lon in the figures.

We will change this in the reviewed version. Note that for lat and lon there is only one B-tree entry, which makes this effectively not visible in the plot. We will clarify this in textual format.

Comparing the metadata size of Table 1 with the kerchunk file size in table 2, can one generally say that kerchunk increases the metadata size by a factor of approximately 2 plus generating these files the inode usage is doubled?

The size of a Kerchunk file is primarily determined by the number of chunks that need to be indexed. Therefore, there is no generally applicable factor by which a Kerchunk version 1 file increases the size, as its size depends on the structure and chunking of the underlying dataset. We will clarify this in the manuscript.

l133: Please explain what uas is (u-component of the near-surface wind). Why did you select a 2D variable for demonstration and not a 3D variable, e.g. ua, which is the 3D wind component? Usually the 3D variables account for the largest fraction of atmospheric netCDF files.

We will clarify the meaning of “uas” (u-component of the near-surface wind) in the revised manuscript. We selected a 2D variable for the demonstration primarily for simplicity and clarity. As discussed above, we expect 2D, 3D, and nD variables, as well as different climate variables such as temperature and precipitation, to exhibit the same B-tree behaviour under a given chunking scheme. The B-tree structure is determined primarily by the number of chunks that need to be indexed, rather than by the dimensionality or physical meaning of the variable. We have also accounted for differences in file size by considering files with different temporal resolutions.

Table3: What is meant by  “Relation of the repacked NetCDF files…”? I rather understand this table as same as Table 1, but for the repacked files. A relation would be, e.g. in the repacked file the number of chunks is only ~2 % and metadata size is only 3.6, 0.8, 0.8 % of the original netCDF files. Please clarify.

We will rewrite this as “Table of the repacked netCDF files...”.

l125: Does cmip7repack work only for CMIP data sets, or can it be applied to all kinds of netCDF files? Are there other tools available that can do repacking? Would it be possible to specify another variable than time to rechunk the data? E.g. thinking about reanalysis data, where each file contains one time step, but many height levels.

The current implementation of cmip7repack uses CMIP metadata conventions to identify the appropriate chunking scheme for the different variables in a netCDF file. Therefore, as currently implemented, cmip7repack is intended for CMIP datasets. However, the underlying approach is not specific to CMIP data and the tool could be readily adapted to other datasets and chunking requirements, since it consists only on a wrapper around h5repack.

l138-141: Could one say that a reduction of the kerchunk file size of more than 98 % is possible? A larger data basis would be helpful to prove this advantage.

As explained above, the size of a Kerchunk file is primarily determined by the number of chunks that need to be indexed. Its size depends mostly on the chunking of the underlying dataset.

Figure 3: Same as Figure 2, the labels are too small to read. Also, please add a sentence to the figure caption that the x-axis is only a fraction of 2 to highlight the changes. Moreover, lat, lon, time are hardly visible. I don't know if they are present in the middle and right panel. In the left panel maybe a logarithmic y-scale could make them better visible.

We will make the labels more visible in the manuscript. For coordinate variables, we acknowledge that they are not visible, due to being only one chunk for these variables. We will clarify this within the text of the figure.

l153-155: Please provide examples for clients that are capable of this.

To the best of our knowledge, Pyfive is the only client satisfying this property, although different configurations of other clients may achieve the same. We will clarify this in the manuscript.

Figures 4 and 5: From how many samples are the values derived? Are they averages? Why are there some tests with relatively large error bars compared to the others? Please provide an explanation. Please consider rearranging the order, to have the results sorted by tool and file format, e.g. kerchunk first, followed by pyfive, and zarr, to facilitate the comparison. Could the error bars also depend on network speed and cloud setup? Would other cloud setups connections show less variability?

We obtained a handful of measurements for each configuration, and all individual results are provided in the GitHub repository. The values shown in Figures 4 and 5 represent the mean for each configuration, while the error bars indicate the minimum and maximum values observed across the samples. We agree that the current ordering of the columns makes the comparison more difficult. We will reorder the results by tool and file format to facilitate comparison, with Kerchunk followed by Pyfive and Zarr.

The relatively large error bars are primarily observed for the dataset-opening times. We consider this variability to be a natural consequence of accessing resources over a WAN, where network conditions and other factors outside the control of the tested approaches can introduce significant variability. We therefore do not interpret this variability as being indicative of the performance of any particular approach. We agree that different cloud and network configurations could exhibit different levels of variability, and we will clarify this limitation in the revised manuscript.

l167-170: Is it just impractical to open the original files or is it even impossible?

The original files can eventually be opened, but doing so requires thousands of HTTP requests and can take up to an hour in some cases. This is therefore not technically impossible, but clearly impractical for a simple dataset-opening operation that should take only a few seconds even in a WAN environment.

Figure 5: I think the clear advantage shown here is that repacking speeds up the reading process significantly. Since the scale is log-scale it would be good to provide some numbers in the text how much faster the variables from the repacked files can be read by pyfive and kerchunk. Isn't Figure 6 a subset from Figure 5? Here the values for “Amon”' are hardly visible. Maybe a table would be the better choice.

We used a logarithmic scale to illustrate the large differences between the approaches, including currently considered cloud-optimized solutions such as Kerchunk. In particular, our results show that repacking the HDF5 files can provide substantial performance improvements, and that applying Kerchunk to the repacked files can be both faster and more efficient than applying it to the original files.

To make these differences more accessible, we provide Figure 6 without a logarithmic scale, allowing the actual performance of the configurations that we consider “production-ready” to be compared directly. This also avoids the visual compression of the differences between these configurations caused by including non-optimal solutions, such as Kerchunk applied to the original, non-repacked files.

l190: Please consider providing an example list of clients that can and cannot fetch chunks concurrently.

We will add some examples, such as netCDF4-python and h5py.

l194-196 & l220-221 & l256-257: The same conclusion is repeated here. Rather have the conclusion just in the conclusions section.

We will rewrite this in the reviewed version.