Ensemble-based techniques have been widely utilized in estimating uncertainties in various problems of interest in geophysical applications. A new cloud retrieval method is proposed based on the particle filter (PF) by using ensembles of cloud information in the framework of Gridpoint Statistical Interpolation (GSI) system. The PF cloud retrieval method is compared with the Multivariate Minimum Residual (MMR) method that was previously established and verified. Cloud retrieval experiments involving a variety of cloudy types are conducted with the PF and MMR methods with measurements of infrared radiances on multi-sensors onboard both geostationary and polar satellites, respectively. It is found that the retrieved cloud masks with both methods are consistent with other independent cloud products. MMR is prone to producing ambiguous small-fraction clouds, while PF detects clearer cloud signals, yielding closer heights of cloud top and cloud base to other references. More collections of small-fraction particles are able to effectively estimate the semi-transparent high clouds. It is found that radiances with high spectral resolutions contribute to quantitative cloud top and cloud base retrievals. In addition, a different way of resolving the filtering problem over each model grid is tested to better aggregate the weights with all available sensors considered, which is proven to be less constrained by the ordering of sensors. Compared to the MMR method, the PF method is overall more computationally efficient, and the cost of the model grid-based PF method scales more directly with the number of computing nodes.
Modern polar orbiting and geostationary airborne instruments provide researchers unprecedented opportunities for remote sensing of the Earth with continuous flows and almost complete spectral coverage of data. The primary cloud retrieval products from satellites are cloud mask (CM), cloud height (CH), effective cloud fraction (CF) and vertical structures of clouds with larger temporal and spatial scales. These cloud retrievals provide an immense and valuable combination for better initializing hydrometeors in numerical weather prediction (NWP), (Wu and Smith, 1992; Hu et al., 2006; Bayler et al., 2000; Auligné et al., 2011) regulating the radiation budget for the planet and understanding the climate feedback mechanism (Brückner et al., 2014; Rossow and Schiffer, 1991; Rossow et al., 1993). Advanced cloud retrieval methods are able to retrieve clouds with multi-spectral techniques (Menzel et al., 1983; Platnick et al., 2003), among which the minimization methods usually directly utilize the difference between the modeled clear sky and the observed cloudy infrared (IR) radiances (e.g., the minimum residual method, Eyre and Menzel, 1989; the Minimum Local Emissivity Variance method, Huang et al., 2004; and the Multivariate Minimum Residual method, Auligné, 2014a). In particular, the Multivariate Minimum Residual (MMR) method is retrieving three-dimensional multi-layer clouds by minimizing a cost function at each field of view (FOV) (Auligné, 2014b; Xu et al., 2015). MMR has been proven to be reliable in retrieving the quantitative three-dimensional cloud fractions with infrared radiances from multiple infrared instruments. However, MMR has limitations in several aspects due to its use of minimization for solution: (1) part of the control variables accounting for the cloud fraction for some certain levels are under-observed since the channels are not sensitive to the existence of clouds for those heights; (2) when clouds at different heights show opacities with the same spectral signal, MMR could lose the ability to distinguish solutions involving clouds at those levels; (3) the computational cost for the minimization procedure in MMR is rather considerable.
Ensemble-based techniques, which usually reside in short-term ensemble forecasting (Berrocal et al., 2007; Shen and Min, 2015), assembling existing model outputs (e.g., cloud retrievals) from varying algorithms (Zhao et al., 2012), or ensemble Kalman filter (EnKF) in diversified forms (Snyder and Zhang, 2003), have been widely developed in order to estimate the uncertainties of various problems in geophysical applications. To better account for the non-linearity between the observed radiance and the retrieval parameter, a novel prototype for detecting clouds and retrieving their vertical extension inspired by the particle filter (Snyder and Zhang, 2003; van Leeuwen, 2010; Shen and Tang, 2015) technique and Bayesian theory (Karlsson et al., 2015) is proposed in this study. As a competitive alternative for MMR, the PF retrieval method has same critical inputs required and cloud retrieval products as in MMR. A brief description of MMR and the new PF cloud retrieval algorithm are provided in the following section. Section 3 describes the background model, the data assimilation system, the radiative transfer models (RTMs) and the radiance observations applied in this study. Model configurations are also illustrated in Sect. 3. In Sect. 4, the single test within one FOV is conducted before the performance of PF method is assessed by comparing its cloud retrievals with those from MMR and other operational cloud products. Section 4 also discusses the computational performance for the two methods. The conclusion and anticipated future work are outlined in Sect. 5.
Essentially, the PF cloud retrieval scheme retrieves clouds with the same
critical inputs requested (i.e., clear-sky radiance from the radiative
transfer model and the observed radiance) and the same cloud retrievals as
outputs (i.e., three-dimensional cloud fractions, which is defined as the
fraction of top of cloud as seen from a sensor) with the MMR method. Both
cloud retrieval schemes consist of finding cloud fractions that allow for best
fit between the cloudy radiance from model and the observation. We use
In this study, a cloud on one model level with a given fraction
The MMR method is an approach to retrieve cloud fractions using the
minimization technique. The residual of the modeled radiance and the
observation is normalized by the observed radiance, which results in the
following cost function, using
Here
Particle filter (PF) approach is one of the non-linear filters for data
assimilation procedures to best estimate the initial state of a system or
its parameters
As the probabilities of the cloud distribution are fully presented by the initial particles, of particular interest is to evaluate different particle initialization schemes in the PF method. Explicitly, the definition of particles corresponds to ensemble members; i.e., one cloud profile as one of particles is corresponding to an ensemble member.
Two approaches for generating particles are initially designed; the first one
is to generate the perturbed samples
Besides those perturbed particles, to represent the existence of a one-layer
cloud on each model level with an even chance, another diversity set of
profiles
A cost function
In Eq. (6), the constraint referred to in Eq. (1) is not respected. Thus, after
the analysis step for the particle filter, the final averaged cloud
fractions
The Advanced Infrared Sounder (AIRS), the Infrared Atmospheric Sounding Interferometer (IASI) and the Cross-track Infrared Sounder (CrIS) are among the most advanced hyperspectral infrared sounders and thus are applied for retrieving clouds with hundreds of channels (Blumstein et al., 2004; Aumann et al., 2003; Xu et al., 2013; Bao et al., 2015; Smith et al., 2015). The Radiance measurements from Moderate Resolution Imaging Spectroradiometer (MODIS) onboard the Earth Observing System (EOS) Terra or Aqua satellites are also well suited to extracting valuable cloud information from the 36 spectral broadband frequencies in the visible, near-infrared and infrared regions at high spatial resolution (1–5 km) (Ackerman et al., 1998). Apart from the IR radiances from polar satellites, the Geostationary Operational Environmental Satellite (GOES) Imager (Menzel and Purdom, 1994) provides a continuous stream of data over the observing domain. In this study, GOES-13 (east) and GOES-15 (west) are also utilized to obtain cloud fractions over the continental United States (CONUS) domain. The GOES Imager used in this study is a five-channel (one visible, four infrared) imaging radiometer designed to sense radiant and solar reflected energy. The instrument parameters for the sensors and the setups for channel selections can be found in Xu et al. (2015).
The background fields are processed running the Weather Research and Forecast (WRF) model (Skamarock et al., 2008). The MMR and PF cloud retrieval algorithms are both implemented based on the Gridpoint Statistical Interpolation (GSI) data assimilation system (Wu et al., 2002) (Kleist et al., 2009), which is a widely used data assimilation system in operations and research in NWP. GSI is capable of ingesting a large variety of satellite radiance observations and has developed capabilities for data thinning, quality control and satellite radiance bias correction. The Community Radiative Transfer Model (Liu and Weng, 2006) (Han et al., 2006) was used as the radiance forward operator for computing the clear-sky radiance and the radiance given overcast clouds at each model level.
The WRF is configured with
The PF experiments apply two groups of particles as mentioned in Sect. 2, among which the group 2 particles contains solely 100 % one-layer clouds. To reveal how the setup of the initial particles impacts the results, apart from the MMR and PF experiments, we included another advanced experiment, denoted as Advanced PF (APF). APF requires more sampled particles including ranges of cloud fractions spanning from 0 to 100 % at intervals of 10 %. An additional experiment “APFg2”, similar to APF but excluding the perturbed particles from the background in group 1 introduced in Sect. 2, was conducted to evaluate the added values from the group 1 particles. In this section, cloud retrieval experiments for several cases containing clouds of a variety of types are conducted for comparison reason. The GOES imager retrieved products from National Aeronautics and Space Administration (NASA; Langley cloud and radiation products) are applied as a reference to validate the cloud retrieving methods for the CONUS domain with a large and uniform coverage of cloud mask. In addition, the retrieved cloud products were also compared to available CloudSat (Stephens et al., 2002) and MODIS level 2 cloud products (Platnick et al., 2003) archived by the CloudSat Data Processing Center in Colorado State and NASA, respectively.
Ratio of the overcast radiances vs. the observed radiance starting from the level 1. The ratio of the clear-sky radiance normalized by the observed radiance corresponds to level 0 (see text for explanation) for the GOES-Imager for channel 5. The approximate pressures corresponding to the model levels are also denoted.
The PF cloud retrieving algorithm retrieves the cloud distributions by
averaging those initial particles with their weights. Before the real case
experiments are carried out over the whole domain, we conduct a single cloud
retrieving test at one FOV to understand what differences can be explained
by the differences in the basic initial particles. In Eq. (5), the
observation error
The weights for different particles with specified cloud fractions
on the
To reveal the roles of various initial particles, Fig. 2a shows the weights
for different particles on the given FOV for channel 5 of the GOES-Imager for
the case shown in Fig. 1. Particles in Fig. 1 include one-layer cloud in
group 2 described in Sect. 2 with specified value of cloud fractions
The normalized
From Fig. 3a, it is found that
The normalized
The retrieval experiments for a real case are conducted at 11:00 UTC, 3 June 2012, when AIRS measurements and the CloudSat “2B-GEOPROF” products
(Mace, 2004) are available. The vertical cross sections of
the cloud fraction field of a real case are illustrated to further check how
different collections of initial particles impact the retrieved cloud
profiles. The standard radar reflectivity profiles from the CloudSat are
shown in Fig. 4a as the validation source; Fig. 4b, c and d
show the cross sections of the cloud fractions along the CloudSat orbit
tracks from the MMR, PF and APF experiments. The vertical structures of the
clouds from MMR compare well with the radar reflectivity from CloudSat by
retrieving the high clouds around 47
The vertical profiles of the averaged cloud fractions from MMR, PF and APF are plotted in Fig. 5 at 11:00 UTC, 3 June 2012, with AIRS. Both MMR and PF experiments yield ambiguous cloud distributions, whereas APF retrieves much stronger cloud signals constrained between level 2 to level 20 (approximately from 950 to 400 hPa). More clouds around level 10 are retrieved (approximately 750 hPa) in MMR, while PF is prone to retrieving clouds near surface levels. Note that MMR retrieves much higher cloud tops and lower cloud bases compared to APF. The cloud base from PF is the lowest; the cloud top from MMR and PF is comparable. Only the APF-related methods will be further discussed in later sections, due to the absence of high clouds using PF.
The mean cloud fraction on all model levels for the experiments MMR, PF and APF with AIRS observations valid at 11:00 UTC, 3 June 2012.
Comparison experiments on real cases are further performed for over longer
time period from 00:00 UTC, 12 December 2013, to 07:00 UTC, 12 December 2013. The
cloud mask is marked as cloudy when there is a recognizable existence of
cloud on any level from MMR or PF retrievals. Both the NASA GOES Imager
products and the MMR-retrieved fields are interpolated to the same
The false alarms, misses and hits for clear and cloudy event
locations with
The retrieved cloud top pressures (CTPs) and cloud bottom pressures (CBPs)
from this study along with the NASA GOES cloud products are illustrated in
Fig. 7. The CTPs from both methods are in good accordance with the NASA
cloud products for high clouds (from 100 to 600 hPa) in Fig. 7a, c and
e. The retrieved cloud top heights from MMR are overall higher than those
from the NASA reference, especially for lower clouds at approximately
750–1000 hPa (e.g., between longitude
The cloud top pressure (left panels) from
The CTPs from NASA GOES cloud products for more hours (03:00, 05:00,
07:00 UTC) together with the independent CTP retrievals from MODIS level 2
products (
The cloud top pressure for
Figure 9 presents the correlation coefficients and biases of the CTPs and CBPs verified against the NASA GOES and MODIS retrievals. The solid lines denote the results regarding the CTPs and CBPs vs. the NASA GOES products from 00:00 to 07:00 UTC, while the dots describe the CTP results vs. the cloud top retrievals in NASA MODIS level 2 products at 03:20, 03:25, 03:30, 05:00, 05:05, 06:35, 06:40 and 06:45 UTC. Here the negative bias means that the retrieved clouds are higher than the reference. Vice versa, the positive bias indicates the clouds are put too low. We conducted another experiment “APFimg” that applies solely GOES Imager data to check the added value from the high spectral resolution radiances (such as, CrIS, AIRS and IASI). In Fig. 9a, the correlations between the retrievals from MMR and the NASA GOES retrievals are comparable with APF for most hours; APF gains overall higher correlations with the CTPs in the MODIS retrievals. From the bias in Fig. 9b, it seems that the CTPs from MMR are underestimated (putting the clouds too high) consistently against both retrievals with GOES and MODIS radiances. Figure 9c shows that the correlations are weaker for MMR compared to others all the time. In Fig. 9d, the positive CBP biases from MMR are remarkable, whereas the CBP biases from APF are largely reduced. Generally, APFimg degrades the CTP and CBP results consistently, suggesting that radiances with high spectral resolutions are able to improve the vertical descriptions of cloud profiles. It is found that the clouds retrieved with APFg2 are shrunken in terms of cloud depth with notably lower cloud top and higher cloud base compared to APF, when excluding the perturbed particles in the first group.
Figure 10a represents the elapsed times for the MMR and APF experiments and the counts of radiance observations in use are shown in Fig. 10b from 00:00 to 07:00 UTC, 12 December 2013. The profile of computing time in MMR is quite different from that in PF. The cost of MMR is dominated by the heavy minimization procedure, whereas APF is more associated with the processes of initializing particles and calculating weights for all the particles. The computing times were measured from cloud retrieving runs with 64 MPI (Message Passing Interface) tasks on a single computing node in an IBM iDataPlex Cluster. The measured wall clock computing times show that generally MMR is computationally more expensive for most of the time than APF. It seems the wall clock times for MMR are generally proportional to the data amount used. While for the APF experiment, the wall clock time is mostly determined by the particles size and partly affected by the channel number, such as for 2013121202 and 2013121206, when the total counts of the hyperspectral sensors (IASI, CrIS and AIRS) are large. The PF experiments using particles of the one-layer cloud with 100 % cloud fractions usually take less than 5 min for the same periods (not shown).
As explained in Sect. 3.3, the filtering problem is resolved in the radiance observational space at each FOV of each sensor independently and sequentially. For each FOV, the retrieved cloud fractions are extrapolated to its neighboring model grid points afterwards. We order the sensors in the cloud retrieving procedure as GOES-Imager, MODIS, CrIS, AIRS and IASI, aiming to optimize the vertical clouds using sensors featured with sufficient spectral resolutions. As a consequence, the retrievals from the last sensor determine the final output to the most extent, causing the cloud retrievals to be highly subjective to the ordering of the sensors. On the other hand, it means the information from other prior sensors will be more or less discarded. In this section, a different way of resolving the filtering problem is preliminarily tested, in which the weights for each particle are aggregated over all available sensors by calling the forward radiative transfer model on neighboring model grids.
Figure 11 shows the clouds retrievals from the grid-based method. It is noted that the grid-based scheme yields slightly worse results of CTP and neutral results of CBP compared with those from the observation-based (FOV-based) scheme, indicating that the hyperspectral sensors probably favor the retrieved CTP and CBP in the FOV-based scheme, which are available for most of the time. It is worth pointing out that the ordering of different sensors has nearly no effect on the final cloud retrievals, when the weights of the particles are calculated in model space (not shown). The final cloud retrieval is no longer overwritten by the retrieval from the last sensor but is a total solution with all the sensors fairly considered, instead. The computational cost of retrieving clouds in model space is comparable or slightly heavier than that in observation space. The computational cost of the grid-based scheme scales with the number of the computing nodes more directly, compared to that of the FOV-based scheme.
This study presents a new cloud retrieval method based on the particle filter (PF) in the framework of GSI, as a competitive alternative to the MMR method. The behaviors of different particle initializations are demonstrated on one single field of view and the CONUS domain. Comparisons between the PF and the MMR method are conducted in terms of the features of cloud mask, cloud top, cloud base and the vertical distributions of clouds. It was found that the PF method retrieves clear cloud signals, whereas MMR is more ambiguous in detecting clouds. By adding more small-fraction particles, high clouds can be better interpreted. From the statistical results, it was found that MMR underestimates the cloud top pressures (put the clouds top too high) and overestimates the cloud bottom pressures (put the clouds top too low) as well. APF improves both the retrievals of cloud tops and cloud bases remarkably, especially for the cloud bases. As expected, radiances with high spectral resolutions contribute to quantitative cloud top and cloud base retrievals. In addition, a different way of resolving the filtering problem over each model grid is tested to aggregate the weights with all available sensors considered, which is proven to be less constrained by the ordering of sensors. Last but not least, the PF method is overall more computationally efficient; the cost of the model grid-based PF method scales more directly with the number of the computing nodes.
In future work, validation studies using multi-spectral imagers on geostationary satellites, spaceborne lidars (or radar) and surface site data will continue, and the results will be used to update the retrieval algorithm. Maximizing the consistency in the products across platforms and optimizing the synergistic use of multiple-source radiances in the new algorithm are important aspects. To estimate the flow-dependent uncertainties in the cloud analysis and in the forecasts, the ensemble nowcasting with three-dimensional cloud fractions via the rapid-update cycling mode is also planned. Increasing the highest extent cloudy cases will be included in future studies. Finally, the use of cloud liquid water and ice mixing ratios retrieved from the cloud fractions using multi-sensor radiances to pre-process the first guess in numerical weather forecast is another promising application.
The MMR cloud retrieval codes can be obtained freely from
(
This work was jointly sponsored by the US Air Force Weather Agency under the project “Air Force Coupled Analysis and Prediction System”, Natural Science Foundation of Jiangsu Province under grant no. BK20160954, the 973 program (grant no. 2013CB430102), the Beijige Funding from Jiangsu Research Institute of Meteorological Science (BJG201510), the National Natural Science Foundation of China (41375025) and the Priority Academic Program Development of Jiangsu Higher Education Institutions (PAPD). The authors would like to thank Chris Davis for fruitful discussions, and to Bobbie Weaver for editing the manuscript. We greatly thank the anonymous reviewers for their valuable comments on the earlier versions of the manuscript.Edited by: S. Remy Reviewed by: two anonymous referees