Global input satellite surface observations were obtained from the Copernicus Climate Change Service (C3S) surface soil moisture COMBINED product v202012, hereinafter referred to as C3S SSM. C3S SSM is a merged product that combines satellite SSM retrievals from 4 active and 10 passive microwave sensors into a daily global dataset on a regular 0.25∘ grid, expressed in volumetric units (m3 m-3) . Invalid retrievals due to frozen ground, dense vegetation, RFI, and other factors are masked out. Although the C3S product provides SSM data from 1978 onward, their quality and spatio-temporal coverage have increased significantly in more recent periods, when sensors measuring in frequency domains better suitable for SSM retrieval became available. Therefore, only C3S SSM data for the period 2001–2020 were used in this study. Note that data from the first year of this period were used only as the model adjustment period and not included in later analyses.

The uncertainty estimates provided for the merged SSM retrievals in the C3S SSM product were computed by means of triple collocation analysis (TCA) . More specifically, (stationary) uncertainties were estimated for each satellite sensor separately and used to calculate the merging weights. Uncertainties in the merged SSM estimates were then calculated from the law for the propagation of uncertainties to account for the quality improvement due to the merging. Note that the distinctive life spans of the satellite missions therefore also lead to distinctive changes in the data quality of the merged product. These sudden changes in product uncertainty are hereinafter referred to as structural breaks . As more and newer sensors provide better-quality retrievals, mean uncertainty values after each structural break typically decrease . This is apparent, for example, in the shift in C3S SSM uncertainty values after the introduction of AMSR-E in 2002 .

C3S data are readily available from the Copernicus Climate Data Store (CDS), and detailed information on the C3S dataset and its underlying ESA CCI v5 merging algorithm can be found in the relevant documentation .

Field measurements for optimizing the model parameters of the EF method and for estimating its uncertainties were obtained from the International Soil Moisture Network (ISMN) for the period 2002–2020 . Only data from sensors with a measuring depth ≤200 cm and internally flagged as reliable were considered. Measuring depths of SM sensors placed vertically in a depth range, e.g. 10–40 cm, refer to their mean measuring depth. Data from multiple sensors installed at the same location and depth were averaged. ISMN data, typically available as hourly readings, were aggregated to mean daily values to match the temporal sampling of satellite observations. Furthermore, we only used ISMN stations where at least 100 data points concurrent with C3S SSM retrievals were available. Notably, approximately 80 % of the selected ISMN time series originate from North America and Europe (Fig. ), and the availability of data declines with depth.

ERA5-Land (E5L) is a multi-decadal climate reanalysis with an extensive portfolio of land variables computed by the assimilation of ERA5 atmospheric variables into the H-TESSEL land surface model . Modelled SM data are available for four depth layers (0–7, 7–28, 28–100, and 100–289 cm) on a regular 0.1∘ grid and are accessible via the Copernicus Climate Data Store (CDS) . We used E5L for a product intercomparison with the RZSM product developed in this study, carried out for the period 2002–2020 within the Quality Assurance for Soil Moisture framework (QA4SM; https://qa4sm.eu, last access: 28 August 2023), which automatically resamples and matches observations of the compared datasets and delivers a wide range of validation metrics.

The EF method relies on a simple two-layer water balance model where the only considered exchange between the surface layer and the reservoir below it is infiltration. The method assumes that the fluxes from the surface to the sub-surface layers are proportionate to the difference in SM content between both layers. In this study, we utilize the recursive formulation of the method : 1RZSM(tn)=RZSM(tn-1)+Kn⋅(SSM(tn)-RZSM(tn-1)), where tn and tn-1 denote timestamps (in days) of the current and previous SSM observations, respectively. Conditions in the root zone are approximated by a weighted combination of the new input SSM observation and past model estimates, with more recent estimates receiving higher weights on a timescale defined by the method's only parameter T (temporal length, typically in days). Weights are controlled by the gain term K, which ranges from 0 to 1 and is calculated as follows: 2Kn=Kn-1Kn-1+e-tn-tn-1T. At initialization, when no preceding estimates are available, the EF calculation is started with K0=1 and RZSM(t0)=SSM(t0).

Temporal variability in the root zone is generally smaller than at the surface; hence the T value and its associated level of smoothing applied to the SSM data increase with depth . The optimal T value (Topt, the value that leads to the best possible representation of RZSM at a certain location using the EF) has been related to differences in utilized SSM sensors , SSM sampling frequency , and land surface features . In particular, T acts as a conglomerate proxy for various environmental factors assumed to govern the infiltration process (e.g. soil texture, evapotranspiration, and climate), but past research on the importance of the exact driving factors is inconclusive and even contradictory . To optimize the T parameter, numerous control factors have been tested , and ever more sophisticated methods have been employed, including machine learning approaches . Other limitations of the method include generally poorer performance in arid zones and when soil texture is not homogeneous throughout the soil column .

Due to the high spatio-temporal heterogeneity of SM and its surface–root-zone coupling – and hence the difficulty in properly estimating the T parameter accurately – an uncalibrated value of T=20 has sometimes been used to describe all of the water content in the first 100 cm of the soil column . Results obtained by using a constant value T=20 were similar to those obtained with T values calibrated for soil texture . Limited sensitivity of the EF to T due to different environmental factors was also observed by other studies, which supports choosing a single value for Topt to represent a particular depth for large areas or even globally .

After filtering out unreliable data (see Sect. ), 3901 ISMN time series from 67 different measuring depths between 0 and 200 cm were available for the T-optimization process. Figure  shows the distribution of Topt values binned into our four chosen RZSM layers (0–10, 10–40, 40–100, and 100–200 cm). The median Topt values for these layers were 6, 15, 48, and 70 d, increasing with soil depth as expected . These median Topt values were then used to compute RZSM globally.

A fivefold cross-validation was performed to verify the robustness of this approach. The variability in median Topt values per soil layer increases with depth but remains negligibly small in all layers, with 6, 15–16, 47–50, and 67–72 for soil layers 1–4, respectively (Fig. a). Subsequently, the five median Topt values derived from the training subsets were used to estimate RZSM for the different layers of the respective validation sets (Fig. c) and resulted in Pearson's r of 0.64–0.67, 0.64–0.65, 0.57–0.6, and 0.48–0.6 for soil layers 1–4, respectively. When evaluating each training set directly, correlations were 0.65–0.66, 0.65, 0.58–0.59, and 0.53–0.56 for soil layers 1–4, respectively (Fig. b).

The little variability between the validation and training sets suggests that Topt values are not over-fitted to ISMN site conditions and can be used robustly in other regions as well. Notably, the spread in median Topt values increases with soil depth, while the correlation scores decrease. This indicates reduced reliability of the method in deeper soil layers, which is in line with the assumption that the coupling between the surface and root-zone SM decreases with depth. Note, however, that results for deeper layers are also affected by the smaller sample sizes at greater depths.

A global SM dataset spanning the period 2002–2020 was computed using the EF method and T parameters optimized at point-scale with the approach described in Sect. . Figure a–e show correlation maps of each of the RZSM product layers as well as the input C3S SSM dataset with E5L; median correlation coefficients are summarized in Fig. f.

The spatial patterns observed in the C3S SSM data (Fig. a) are strikingly similar in RZSM layer 1 (Fig. b), with observed slight to moderate deterioration in performance over the high latitudes (>60∘ N). This is not surprising given that both products differ only by a small degree of smoothing applied to RZSM layer 1 and are compared to the same E5L layer (0–7 cm). RZSM layers 2 and 3 (Fig. c–d), respectively, are compared to E5L layers 7–28 and 28–100 cm, respectively, and largely preserve good performance in regions where the input C3S SSM product also performs well, i.e. Europe (bar Scandinavia), the Caspian and Aral sea basins, the eastern United States, India, Southeast Asia, South America, sub-Saharan Africa, and Australia. At the same time deterioration of performance is observed at high latitudes and in arid environments such as the Sahara desert and the Arabian Peninsula, where the reduced strength of coupling between the surface and root-zone dynamics hinders the EF performance . Areas of good and poor performance visible in RZSM layers 1–3 are not apparent in RZSM layer 4 (Fig. e), where agreement with the reference E5L is very heterogeneous. A few regions where the good performance observed in shallower layers was preserved are India, Southeast Asia, and the eastern United States.

The median Pearson correlations between the RZSM product and the E5L reference layers (0–7, 7–28, 28–100, and 100–289 cm) were 0.54 (RZSM layer 1), 0.47 (RZSM layer 2), 0.41 (RZSM layer 3), and 0.28 (RZSM layer 4), respectively (Fig. f). The C3S SSM data included in this comparison show only marginally better performance against the E5L reference (0–7 cm) than the RZSM layer 1, with a median Pearson correlations of 0.55 and 0.54, respectively.

The results are also consistent with the assumption of the EF model that SM dynamics decrease with depth and that Topt ought to increase accordingly, as was also found by other studies . At the same time, the maximum correlation values decrease with depth, confirming the diminishing coupling between the surface and root-zone layers, as also found at the in situ station level (Fig. ) and demonstrated by others .

Figure  shows estimates for the model structural uncertainties (Sect. ) obtained at all available in situ sites, binned into the four RZSM product layers. Their median values (represented by orange lines and annotated) were used as estimates for σ(EF). Note that in situ measurement errors were assumed to be negligible and thus did not influence ubRMSD estimates, which likely causes model structural uncertainties to be overestimated. Also, structural uncertainties are assumed to be constant in time.

As anticipated, an increase in σ(EF) corresponds to the growing distance between the surface and the root-zone measurements, demonstrating the decreasing coupling strength between both layers. Note that σ(EF) shows significant variability within RZSM layers, which is likely, at least to some degree, related to variations in local conditions. However, as with the T-parameter optimization, we estimate structural uncertainties based only on a limited number of in situ stations and therefore use the median to extrapolate globally.

Figure  compares RZSM uncertainty estimates obtained from the baseline method with those from the approach proposed here. Figure a shows a time series of RZSM uncertainties from the baseline method at an arbitrary location. Figure b shows the effect of changing σ(T) from 10 % of Topt to the median absolute deviation of Topt, which is an amplified temporal variability. Simultaneously, mean uncertainty values increase and become closer to the magnitudes of the input SSM dataset. Moreover, they no longer diminish with increasing T values (i.e. depth), as is the case in the baseline formulation. This is presumably more realistic since the progressive decoupling between the surface and deeper soil layers can be expected to cause uncertainties to increase rather than to decrease.

Figure c shows the impact of accounting for σ(EF) in the total uncertainty budget when using 10 % of Topt as the T-parameter uncertainty (σ(T)). Considering this term substantially increases the magnitude of the propagated uncertainties and leads them to increase with depth (as does σ(EF)). However, the uncertainties' temporal variability is reduced substantially as the effect of σ(T) is overshadowed by that of σ(EF). Finally, Fig. d shows the combined effect of using the MAD of Topt as its parameter noise σ(T) and accounting for model structural uncertainty σ(EF). Compared to the baseline (Fig. a), this yields an increased overall magnitude of the uncertainties; a more realistic increase in (temporal average) uncertainties with depth; and an amplified temporal variability in all layers during transitions between dry and wet conditions, which is also expected (see Fig. ).

Similar to , we assess the use of the proposed MAD estimates for σ(T) by computing Pearson's r and root-mean-square differences (RMSDs) with respect to in situ data before and after removing a fixed percentage of the data (5 %, 10 %, 15 %, and 20 %) with the highest uncertainty estimates. In the case of effective correspondence between high values of both the estimated RZSM uncertainties and the observed RZSM deviations from reference in situ measurements, it is expected that the skill metrics will improve due to the masking. This hypothesized correspondence holds well as long as the difference between in situ and satellite-based RZSM values is mainly due to the random errors in the latter. Note that this analysis is only sensitive to the impact of using different values for σ2(T) ((Topt/10)2 versus MAD(Topt)2) since the estimated structural uncertainty σ(EF) is constant in time and therefore cannot change the ranking of the total uncertainties.

Figure a and d indicate (in magenta shading) 20 % of RZSM layer 2 data with the highest uncertainties masked out in the experiment described above, based on uncertainties estimated with the baseline (b) and our method (d), respectively. Overall, despite the differences in magnitude and amplitude, both our and the baseline method assign the highest uncertainty values to timestamps corresponding to significant soil wetting or drying events. However, with the baseline method the average magnitude of SSM input uncertainty appears to have a greater influence on the calculated RZSM uncertainty estimates. This is most evident when comparing values before and after the inclusion of Metop-A ASCAT into the C3S product in January 2007 (indicated by the dashed vertical line in Fig. ), which substantially improved data quality thereafter. Specifically, the mean C3S SSM uncertainty dropped from 0.029 to 0.018 m3 m-3. Such a clear shift is also visible in the uncertainty values propagated with the baseline method (from 0.008 m3 m-3 before to 0.004 m3 m-3 after the introduction of Metop-A ASCAT). This causes the baseline method to predict that the majority of the 20 % most uncertain SM values will occur in the pre-ASCAT period. In contrast, in our approach, average uncertainties remain stable (at 0.036 m3 m-3) over the entire time period. This suggests that the use of MAD(Topt)2 as an estimate for T-parameter uncertainty reduces the sensitivity to structural breaks, i.e. large variations between the uncertainties in the C3S SSM input sensors, and improves the method's capability to predict day-to-day uncertainty variations. Lastly, after the introduction of ASCAT, both schemes consistently assign higher uncertainties to timestamps characterized by large SM changes. Taken together, while the use of Topt/10 as the T-parameter uncertainty seems to yield realistic estimates for uncertainty variations due to the use of different C3S SSM input sensors, using MAD(Topt) as the T-parameter uncertainty seems to better predict day-to-day uncertainty variations in the RZSM estimates.

Figure  shows the results of the data removal experiment described above, summarized for all considered ISMN stations. To compare the performance with and without the effect of C3S structural breaks on the uncertainty values (see above), results are shown for both the full product period (2002–2020; Fig. a–d) and a sub-period without breaks, i.e. from the inclusion of SMAP data onward (1 April 2015–2020; Fig. e–h). In both cases, correlation coefficients obtained for the complete time series were compared to those obtained after removing 5 %, 10 %, 15 %, and 20 % of data with the highest associated uncertainties.

In the case of the full product period (Fig. a–d), using σ(T)=Topt/10 as the T-parameter uncertainty seems to consistently yield more realistic estimates of temporal uncertainty variations than using σ(T)=MAD(Topt). This is true for all four soil layers. Masking out more uncertain data indicated by either method consistently improves agreement with in situ reference data in the first two product layers. This improvement increases the more data are masked out, as is expected. In the absence of such breaks (Fig. e–h) RZSM uncertainty variations seem to be better predicted when using σ(T)=MAD(Topt) as the T-parameter uncertainty in almost all cases. Notably, in layers 3 and 4, data removal according to either method degraded the agreement with field measurements.

In summary, the propagation of C3S SSM input uncertainties yields accurate predictions of temporal uncertainty variations in RZSM estimates obtained with the EF method for the first two layers (0–10 and 10–40 cm). This is no longer the case for deeper layers (40–100 and 100-200 cm). Note, however, that the RZSM estimates in these layers themselves still exhibit reasonable skill when evaluated against E5L (see Fig. ).