The Southern Ocean (SO) shortwave cloud radiative bias is a well-documented problem in global climate models as well as some numerical weather prediction models . The bias is characterised by too much shortwave radiation reaching the surface of the ocean and not enough being reflected by clouds back out to space. Significant work has been done to identify the cause of this model bias. and have shown that a large part of this bias can be attributed to the inability of models to simulate super-cooled liquid water clouds in the SO, in particular in cold sectors of extra-tropical cyclones. A number of observational studies have shown the prevalence of super-cooled liquid water clouds over the SO . The prevalence of super-cooled liquid water clouds is attributed to the pristine conditions found in the region removed from the sources of terrestrial (e.g. dust/biomass burning) and anthropogenic (e.g. black carbon) aerosol (although these aerosols can occasionally intrude into the region). The lack of these particular aerosols, which contribute large sources of ice-nucleating particles (INPs), limits the ability of cloud droplets to freeze, resulting in liquid clouds at temperatures well below zero. Currently, many climate and weather models do not take into account the pristine composition of the SO atmosphere, assuming, like much of the world, that INPs are available to help freeze cloud droplets, resulting in too many ice-phase clouds which allow too much shortwave radiation to reach the surface of the ocean .

Numerous studies have attempted to address the shortwave cloud radiative bias via cloud-phase parameterisations, including but not limited to treatment of ice-nucleating temperatures , ice crystal shapes and growth rates , INP number concentrations and sources , convective detrainment temperatures . Invariably, many of these studies find that altering parameters for specific SO conditions results in changes in model performance over other parts of the climate system for better or worse as explored in. This outcome is particularly important given the range of recent literature highlighting a latitudinal dependence of cloud properties in the Southern Ocean, which has been attributed by some to differences in aerosol properties .

Many model evaluation techniques used to diagnose this problem rely on satellite observations and hence require a satellite simulator such as the Cloud Feedback Model Intercomparison Project (CFMIP) Observation Simulator Package (COSP) described in . More recently, ground-based lidar simulators have also become available for more accurate model evaluation from ceilometers and lidars on the surface e.g.. In conjunction with such simulators, most studies have separated their data into specific conditions, for example, by isolating particular synoptic situations e.g. or by cloud regimes (or “weather states”) e.g.. By isolating particular conditions, the specific microphysical causes of the radiative bias that are relevant to that condition can be diagnosed. However that cause may not perhaps be relevant to other conditions. Additionally, as (a companion paper to this study; herein F22) found, the model being examined in this study rarely simulates cloud regimes correctly, which may reduce the usefulness of the cloud regime approach and calls for a different method of evaluation. Without the use of synoptic typing or cloud regimes, calculating zonal means are a popular way of diagnosing model biases at a macro-scale. However, this method severely limits the microphysical inferences that can be made.

Another way to evaluate and in some instances tune models is to explore parameter uncertainty . In these cases, the parameter space (the range of plausible values) and its impacts in global climate models are emulated with more simplified statistical models. This allows re-sampling over a range of multi-parameter values many times over what is possible with physically driven models. From these large samples, the uncertainty attributed to particular parameters can be identified, and the best combination of parameter values can be constrained based on comparisons with observations. These methods present a powerful way of reducing uncertainty in climate models within known and quantified parameters and physical mechanisms.

The evaluation techniques presented in the literature are important methods in understanding model biases and have been shown to be useful in testing and tuning new parameterisations. However, these techniques are often limited to using human ability to discern complex physical processes, interactions and patterns to diagnose the drivers of biases. We suggest that utilising machine learning and associated feature importance metrics can enhance pattern recognition, aid our ability to assess non-linearity and collinearity, and shed new light on our understanding of the underlying causes of the biases across multiple conditions.

Increased computing power as well as increased data availability now means that machine learning techniques are useful tools to further understand and predict climate and weather problems, especially in relation to clouds . Currently, applications of machine learning in climate science are limited for a number of reasons, including the relatively recent advances in the field with respect to both methods and accessibility of computing resources, the difficulty in applying often non-perfect or very limited datasets to a problem, and the fact that physical understanding in climate science is often more important than model accuracy . Current applications include (but are not limited to) predicting a particular field, such as low marine clouds , liquid water path and radiation ; improving retrievals for remote sensing ; downscaling of coarse-resolution data ; improving subgrid-scale parameterisations ; or classification problems .

Of particular note is the application of machine learning to climate emulation, i.e. emulating the global response of complex climate models, as outlined in . Climate emulation has typically used simple models to estimate what the response of the climate (usually temperature) may be to changes in forcings. These models tend to not capture spatially varying and non-linear processes well, whereas machine learning has been shown to do well in this space but has been challenged by a lack of data for training purposes. have now provided a dataset and some initial machine learning frameworks designed specifically for training models for this application, which may provide a new way to determine possible climate responses to changes in forcings beyond that of the temperature.

To our knowledge, no study has yet applied machine learning methods to understanding biases in climate models. We now believe that current methods for regression problems combined with feature importance metrics may provide useful insight into how biases are occurring. Feature importance metrics aim to provide quantification of how each predictive feature in a problem (regressive or categorical) has contributed to the final outcome. Commonly, feature importance metrics have been considered misleading due to their inability to take into account dependencies between predictive features . However, the recent advances in this space, including the development of the SHAP (SHapley Additive exPlanations) feature importance, mean that we can better take into account these dependencies.

SHAP analysis builds on Shapley values, a method originally derived for game theory applications to identify how important one player in a team was to the outcome of the game . SHAP feature importance can be used to assess individual, accumulative and interacting feature importance, taking into account collinearities. Furthermore, SHAP analysis is model agnostic and is considered a powerful tool in feature analysis.

In this study, we combine SHAP feature analysis with a regression model to evaluate and understand cloud radiative model biases for the first time. We test if we can perform such an evaluation in a more holistic manner than in previous studies, considering all conditions at once rather than specific cloud or synoptic regimes. To do this, rather than isolating a particular regime and then examining the particular biases in cloud properties associated with it, biases in cloud microphysical properties are used to predict the bias in the cloud radiative effect. We then apply SHAP feature importance to understand the primary drivers of the cloud radiative bias at any point in space or time. We hypothesise that this method can provide new insight into the cloud radiative bias problem and may be useful tool when it comes to model sensitivity testing.

Figure a shows the difference in December–January–February (DJF) SWCRETOA between the ACCESS-AM2 and satellite product (CERES-Syn1Deg). As a reminder, the SWCRETOA is calculated as the clear-sky radiation minus the all-sky radiation fields, which results in negative SWCRETOA values (see Fig. ). The biases are calculated as model minus observations, with positive SWCRETOA biases indicating that the model has less negative values than the observations, resulting in a positive SWCRETOA bias, indicating that less sunlight is being reflected out to space.

As discussed in F22 , a strong bias in the polar region of the SO is found, corresponding to too little shortwave solar radiation being reflected back out to space by clouds and too much being absorbed into the Earth system, including reaching the surface of the ocean. In the sub-polar region, a zonally asymmetric bias is found with positive biases shown in the eastern Indian Ocean and Pacific Ocean sectors, while negative biases are found throughout the rest of that region as well as in the mid-latitude region. Examination of the model and satellite fields separately (see Appendix Fig. ) shows that the asymmetrical bias appears to be due to ACCESS-AM2 failing to capture the observed spatial variability. We will consider causes of this asymmetry again in Sect. 4.2.

The spatial variability in this bias suggests that cloud/radiative regimes strongly vary across the Southern Ocean. The asymmetry of this bias makes it difficult to evaluate and understand without splitting the region up into specific synoptic or cloud regimes, as done in studies such as or . Building on knowledge from previous work, where we understand that particular cloud characteristics (or lack of) are responsible for driving the SWCRETOA bias in parts of the Southern Ocean, in this study we use biases in such cloud characteristics to predict the SWCRETOA bias. We hypothesise that if we can satisfactorily predict a bias, we can then use the derived XGBoost model to better understand the sources contributing to the bias. Firstly however, we want to understand more about the individual relationships of each cloud feature bias and the SWCRETOA bias.

Correlations have been calculated for each cloud feature (shown in Fig. ) with respect to the SWCRETOA bias as well as to each other to test for both linearity of the cloud radiative bias relationship and collinearities between the cloud features. We find that the LWP bias has the strongest relationship with the SWCRETOA bias but can only linearly explain 30 % of the variance. The other cloud features explain very little of the SWCRETOA bias under a linear assumption. Similarly, explained variance between cloud features does not exceed 31 %, indicating weak collinearity. Analysis of the mean SWCRETOA bias versus the cloud feature biases for each of the cloud types developed by F22 over three latitudinal areas is shown in Appendix Fig. . This figure confirms, even when only considering the means across cloud types/locations, that these relationships are in some cases highly non-linear and in other cases very weak.

In reality, how clouds interact with radiation is a complex system that depends on many, not individual, variables. Hence, analysis comparing singular features should not be expected to provide strong indications of how this system works. For this reason, we also wanted to test whether a simple technique such as multiple linear regression (MLR) could predict the SWCRETOA bias more satisfactorily. The benefit of such a technique is the low computational requirements and easy initial interpretation. The MLR was able to predict between 42 %–43 % of the variance (when tested on different summers, in the same way as described for the XGBoost training and testing datasets). We have provided more detail on how the MLR prediction performs in the Supplement. The improvement of the MLR compared to a linear prediction from the individual cloud features does give us an expectation that we can improve upon this problem with a more sophisticated tool that can account for the inherent characteristics of the data.

The tool we have chosen to address these issues is XGBoost because it can handle non-linear applications and its performance is not significantly impacted by collinearity among predictor variables, although as we discuss shortly, these must be still considered when we interpret the importance of these predictors. Using XGBoost, we model the DJF SWCRETOA biases using the biases in the cloud features discussed above. The results for each period of training and testing were similar, predicting between 54 %–55 % of the SWCRETOA bias (the R2) and a root mean squared error between 29.45–30.12. For the subsequent results and analysis, we use the full dataset, where 58 % of the SWCRETOA bias is explained.

Figure  shows the XGBoost-predicted bias (b), the residual between the true and predicted bias (c), and a histogram of the residual versus the predicted bias (d). If we were to consider just the median values of the entire region, the XGBoost model predicts a SWCRETOA bias of 11.4 W m-2 compared to 11.7 W m-2 (the means are 12.4 and 12.4 W m-2, respectively), implying that the XGBoost model performs quite well, albeit with a lower standard deviation (32.9 W m-2) than that of the true bias values (44.4 W m-2). The area-weighted statistics for the entire region for the true and XGBoost-predicted values, respectively, are as follows: means of 12.0 and 11.5 W m-2, medians of 11.1 and 10.6 W m-2, and standard deviations of 45.2 and 33.5 W m-2. In Fig. d, a more symmetrical concentration of residuals (y axis), centred around zero, and a narrow range of predictions are indicators of a well-performing model (x axis). We can see that the XGBoost model does provide a relatively symmetric pattern, with little skew in any direction. This is especially the case when compared to the MLR, shown in the Supplement.

We get a much clearer picture of the XGBoost model's performance in different spatial regions in Fig. a, b and c. Here, we can see that the XGBoost model appears to capture the SO negative bias in the polar region reasonably well, with the predicted median of 23.2 W m-2 and a true value of 19.4 W m-2. The residual is fairly uniform except in the Weddell Sea region, which could reflect the influence of sea ice. In the sub-polar region, there are small differences between the predicted and true SWCRETOA biases, with a median predicted bias of 6.9 W m-2 compared to the true bias of 7.3 W m-2. Interestingly, these small residuals are not zonally symmetrical, with positive values in the Australian and Pacific sectors and negative biases in the Atlantic and most of the Indian Ocean sectors. In the mid-latitude region, the positive SWCRETOA bias is slightly underestimated in the XGBoost model (median of 8.6 W m-2 for the predicted bias compared to 12.6 W m-2). The residuals, however, are more zonally symmetric than the sub-polar and polar latitude bands.

As stated previously, our XGBoost model can explain just over half of the variance in the SWCRETOA bias (R2=0.55). The remainder of this variance, as well as some of the spatial differences observed in Fig. c, suggests that the biases in the cloud fields used in this work may not be the only contributors towards this particular feature of the radiative bias. Alternatively, it could also imply that the robustness of the data that we consider truth are not the same for all regions. The study that the current work follows on from, F22, showed little zonal asymmetry in the LWP, IWP, CFL and CFI biases and similarly little asymmetry in the relative frequency of occurrence biases of different cloud types, suggesting the need to account for additional physical properties and relationships in these regions.

The missing component could be a range of things, for example, another microphysical cloud field, dynamical field or environmental factor, i.e. the presence of INPs, sea surface temperatures or thermodynamical properties of the atmosphere. For example, the influence of dust-derived INPs may be greater in particular sectors of the SO in certain seasons, which may change cloud properties in those regions. Alternatively, , in predicting LWP over marine boundary layer clouds in the southeast Atlantic, found that dynamical characteristics and sea surface temperature were important environmental factors along with other cloud property predictors.

Although the XGBoost model is able to explain a larger amount of the SWCRETOA bias than any individual or linearly associated cloud feature could, we must keep two things in mind. Firstly, by adding more (and in particular, not internally consistent) data we make interpretation more difficult, and secondly, the relationships shown here do not prove a causal relationship. We must understand what the model is inferring physically for us to make such an association. In this work, we show that even with only 54 %–55 % of the variance explained by these cloud biases, we can derive valuable information from how they contribute or do not contribute to the radiative bias. For these reasons, we will continue with our described method and explore how each of these cloud features contributes to predicted SWCRETOA bias using SHAP analysis.

SHAP feature importance analysis allows us to quantify the contribution (or importance) each cloud feature made to the predicted result, in this case, the XGBoost radiative bias. This is demonstrated in Fig. a, where a so-called force plot is shown. Here, we can see how, for this singular prediction of x, each cloud feature has contributed to the total prediction f(x), by shifting the values away from the base value (12.4 W m-2). We note that for the cloud features, their actual values, not the SHAP values, are shown in text for the top predictors, while the magnitude and sign of the SHAP value are indicated by the bar. The base value can be considered the starting point of any prediction; i.e. if we had no information about the cloud state, a good prediction to start with would be the base value. Each cloud feature then adds subsequent information to the prediction, which all together adds up (with the base value) to the total prediction. This also means that when summed together for any individual point, or subsection of points, the SHAP values do not represent the total prediction for that point or subsection but rather the difference from the base value. This is demonstrated by Fig. b, which shows the mean sum of all SHAP values spatially, which are equivalent to the difference between the predicted bias (shown in Fig. b) minus the base value. This characteristic of the SHAP analysis must be kept in mind when considering the SHAP values.

We have demonstrated the usefulness of the SHAP method in quantitatively understanding drivers of biases. This method has provided useful insights into the mean drivers of the SWCRETOA bias as well as into cloud-specific influences. We now want to test if this method is useful in understanding how perturbations applied to a model may change the results. To do this, we have performed a second nudged simulation where we have altered the ice crystal capacitance following the work of . We refer to the new simulation as the “ice” simulation (compared to the “control”). In the ice simulation, we have changed the capacitance from 1 in the control, which assumes a spherical crystal, to 0.5, which assumes an oblate ice crystal. The effect of this change is to slow down ice crystal growth rates. Overall, reducing the ice crystal growth rates reduces the radiative bias by -1.0 W m-2 for the entire region (-1.2 W m-2 for polar region, -1.0 W m-2 for sub-polar region and -0.8 W m-2 for mid-latitudes) in our simulations. This difference is smaller on average and noisier than that found in (and not statistically significant), which we expect is a result of the nudging of the model. Similar constraints on meteorology were found in when nudging was applied during model perturbation experiments. This study compared perturbed nudged simulations to perturbed free-running simulations. The nudged simulations had a smaller overall response with a lot of seemingly random variation, which we refer to as “noise”. Regardless, this small perturbation experiment provides an opportunity to test the usefulness of the SHAP method, even if the changes are very small.

With this second simulation, we have re-trained (and re-tuned) our XGBoost model. The performance of the XGBoost-ice model is very comparable to that of the control, explaining between 54 %–55 % of the variance. The mean difference in the predicted bias from the control is similar to that of the true bias of 0.9 W m-2. Once again, LWP is the most important cloud feature, and the order of importance for each cloud feature has not changed.

Figure  shows the changes in SHAP values (ice-control) for each cloud field along the top row and the changes in the fields themselves along the middle row (also ice-control). The noisiness introduced by the nudging is clear in these plots, and we note that none of the changes are statistically significant. Nevertheless, some interesting features can been seen in these results.

Firstly, we can see that changing the ice crystal shape, which slows ice growth and increases LWP in the polar and sub-polar regions (Fig. h), reduces the overall bias in the polar region (which is negative – see Fig. h). In the mid-latitudes, the change is much less coherent. The resulting change in SHAP values indicates that in the polar region, the increased LWP has overall reduced the SHAP values, although results are quite noisy. Figure o demonstrates that polar and sub-polar stratocumulus clouds are associated with the largest increase in LWP and reduction in SHAP values. Interestingly, shallow cumulus clouds, despite showing only small increases in LWP, are also associated with larger SHAP values for all regions.

CFL (Fig. i) shows a varied and very small response to the change in ice crystal growth. CFI (Fig. j), on the other hand, shows a clear and consistent increase in fraction. This increase in CFI may be caused by an increase in cloud lifetime as the ice crystal growth rate slows down, which is supported by a small decrease in snowfall rates over the polar region (not shown). The change in SHAP values for CFL is predominantly negative in the polar regions and positive in the sup-polar to mid-latitudes. For CFI, the changes in SHAP values are harder to differentiate.

Considering these changes are grouped by cloud type, some interesting patterns are apparent, though we note the magnitudes of these values in cloud fraction are again very small. For CFL, higher clouds (cirrus, convective and frontal) are found to be associated with a reduction in CFL and an increase in SHAP values. In the case of mid-latitude convective clouds, we suggest this is caused by the associated increase in LWP converting into an increase in rainfall, resulting in an overall reduction of cloud lifetime. For the remaining cloud types, the picture is less clear with very little change in rainfall. Polar marine stratiform and cloud decks exhibit a stronger positive change in CFL associated with a decrease in SHAP values, while the remaining cloud types (for all regions) show a much more neutral response. For CFI, all cloud types, for all regions, experience on average an increase in fraction, but this increase results in a positive change in SHAP values for the lower, thicker clouds and a negative change in SHAP values for the higher, thinner clouds. The varying behaviour of higher/thinner clouds compared to lower/thicker clouds for CFI and CFL, with respect to both the cloud properties themselves and their resulting SHAP values, highlights that a “one-size-fits-all” approach to reducing cloud radiative biases in models is not appropriate.

We will not discuss the changes in optical depth or IWP for brevity. We will, however, touch on CTP in Fig. g, n and u. Consistent reductions in CTP are found for the entire region, which act to increase in magnitude the already negative bias (Fig. n). The changes to SHAP values on average are small. However, if we consider these values separated by cloud type/region, we can see a strong separation in behaviour between cloud types. Thinner, higher clouds, which were on average overestimating CTP, are found to have larger decreases in CTP. The lower, thicker cloud types exhibit a smaller magnitude decrease, only minimally exacerbating the CTP bias (see Fig. u). These two groups of clouds types are also associated with different SHAP values, where the lower/thicker clouds decrease in CTP (increase in cloud height), resulting in more positive SHAP values. The higher/thinner clouds, despite also decreasing in CTP, are found to have negative changes in SHAP values.

The results presented in this section indicate very small, insignificant changes to cloud properties when ice crystal growth rates are slowed within a nudged model framework. Nevertheless, these changes have resulted in the expected decrease in radiation bias and are of use to us within the context of this analysis: can we use SHAP values to examine changes in model parameterisations? Here, the behaviours of cloud types/regions, with respect to changes in cloud properties/SHAP values, are of greater interest to this work for a number of reasons. Firstly, these results highlight that despite the averages presented in the top two rows of Fig. , cloud radiative interactions are much more complex. Importantly, this analysis has been able to condense and analyse a large amount of information in a more succinct way than previously possible, with many prior studies only concentrating on one region and/or one cloud type (typically low-level stratiform). Secondly, this analysis clearly demonstrates instances of compensating radiative errors between different cloud properties, types and regions. While spatial compensating errors have previously been identified, few prior studies have examined them with respect to cloud type. To our knowledge, no other study has been able to directly associate changes in cloud properties to changes in the radiative bias (though we stress that the results here do not present a casual relationship).

We believe that this last point is where the power of SHAP analysis lies. In this work we can see that, for example, for low/thick clouds in the polar region, the resulting changes in LWP, CFL and IWP caused by the model perturbation work to decrease the prediction of the radiative bias, while CFI and CTP have the opposite effect. Our method is able to easily account for individual and interacting, sometimes non-linear, influences from each predictor, allowing us to interpret our predictions with confidence.

The SO radiation bias has been a topic of considerable research over the last decade, motivating a number of innovative methods and studies to understand its controls. While methods such as cloud regime clustering or cyclone tracking have become standard ways to evaluate the SO radiative bias e.g. as in, they do not account for the entirety of the SO and the range of biases found across it. This narrower view has proven detrimental to some model development studies, where some aspects of cloud microphysics, such as ice nucleation temperatures, have been altered to target the worst of the radiative bias but have led to unwanted changes in other regions e.g. as in.

In this study, we have proposed a new method for model bias evaluation, employing modern machine learning and taking advantage of the large amount of cloud and radiation data available to us and facilitated by enhanced computing resources. This method, where we use biases in cloud properties to predict the SWCRETOA bias, considers the entire SO, from the mid-latitudes to the polar regions, at a daily timescale. This study has been made possible due to our ability to perform a nudged climate model simulation, an underutilised method in climate research without which coincident in-time comparisons to satellite fields are not possible. This work provides a new perspective on the downwelling shortwave radiation bias, which will help guide our future model developments. Furthermore, we expect that this method could be applied to a range of complex model biases throughout the climate system beyond radiation.

We note that the cloud fields used in this work, including the satellite products and the modelled products, each contain inherent uncertainties. While the MODIS L3 product has specifically been produced for model evaluation, we must acknowledge that these products may not represent the “truth”. Greater discussion on this can be found in F22. Similarly, find an underestimation of shortwave cloud radiative effect at the surface of 7.9 W m2 in CERES compared to ground observations at Macquarie Island, indicating that similar issues exist in the satellite radiative fields. To add to this, despite satellite simulators such as COSP being designed to reduce the uncertainties between modelled and satellite-retrieved products, we have found that some of these simulated fields were of too poor a quality to be used with confidence. This was the case for the LWP and IWP fields, for which we used the raw modelled products instead. While this decision adds to the uncertainty in our analysis, we are nonetheless confident in our overall results (e.g. LWP being a dominant driver of biases radiative biases).

Specifically, in this work we have continued our evaluation of the ACCESS-AM2 model's SO cloud and radiative biases from. The ACCESS-AM2 SO SWCRETOA bias is shown to be the largest in magnitude over the polar region, while it weakens in the sub-polar and mid-latitude regions. Of interest to this work in particular is that the ACCESS-AM2 model lacks the zonal asymmetry of the CERES-Syn1D SWCRETOA, with positive biases in the Australian and Pacific sectors. This asymmetry has not previously been considered and is possibly a reflection of the differences in SWCRETOA biases between cloud regimes or other unaccounted for physical processes.

Importantly, the XGBoost model suggests that the ACCESS-AM2 SO SWCRETOA bias cannot be completely explained by the biases in several key cloud properties, including LWP, IWP, CFL, CFI, TauL, TauI and CTP. Many of these cloud biases have a non-linear relationship with the radiative bias as well as weak collinearities amongst them, demonstrating the complexity of the system and the need for a method that can take such complexities into account. For this purpose, we have used a tuned and tested XGBoost model to predict the SWCRETOA bias, using the biases from these cloud fields as predictors.

The XGBoost model can explain up to 55 % of the SWCRETOA bias, more than any one of the cloud fields alone or a linear combination could. While the general pattern of the radiative bias is captured, the zonal asymmetry is not, with the XGBoost prediction lacking the positive values found in the sub-polar region. This finding suggests that the asymmetry may not be a function of the cloud properties explored in this work but possibly some other environmental factor, such as a dynamical or thermodynamical property, aerosol sources and interaction, or proximity to the polar front. Other studies have found that environmental factors, such as SSTs or dynamical predictors, are important for predicting cloud characteristics or radiation . Such factors may contribute to the missing 45 % of predictability, though this has not been tested in this work. For this analysis, we have chosen not to explore additional parameters for three reasons: (1) to maintain interpretability – the more parameters you have the less interpretable your model becomes; (2) to maintain some level of data homogeneity – we try to limit the number of sources (e.g. different satellite products) our predictors are coming from to limit inconsistencies in assumptions; and (3) to keep our focus on the cloud radiative relationship, without compounding it with other external factors. However, we suggest that future work should explore how environmental factors may influence the SWCRETOA bias, including looking at cloud condensation nuclei (CCN) availability, cloud droplet number concentration, INP concentration, sea surface temperature and vertical cloud overlap.

One of the benefits of using the XGBoost model is that it can efficiently be interpreted and analysed with the SHAP feature importance method. Our SHAP analysis has shown that the biases in LWP are the main drivers of the SWCRETOA bias (with CFL, CFI and IWP following). Further exploration of the SHAP feature importance and the cloud biases indicates that the relationships are still non-linear, but for the most part, we can make physical sense of them. In addition, we have shown that there are cloud-type-specific behaviours that can be easily captured using this type of analysis, including that for mid-latitude and sup-polar mid-level stratocumulus and marine stratiform clouds, CFL has higher SHAP values than LWP, indicating a greater importance. This method allows us to evaluate cloud radiative biases in a much more holistic way compared to isolating just one, often pre-designated, “important” cloud type. Furthermore, we find distinct cloud-type behaviours in the SHAP–bias relationship, which we expect can be leveraged in future model development.

We have tested this by exploring a simple model perturbation, previously described in , where we reduce the ice shape capacitance from 1.0 to 0.5, thereby reducing the ice crystal growth rates, increasing LWP and reducing the radiative bias by approximately 1 W m-2. We note that saw much larger radiative changes in response to this perturbation, which we suggest is due to the free-running nature of their experiments. Although the changes between this test run and our control were small (and insignificant), we believe that our results demonstrate the power of SHAP analysis, where complex changes within a system can be evaluated and their impact quantified.

This finding is particularly relevant to methods used to constrain models to observationally plausible values, such as that done in . In these studies, perturbed parameter ensembles (PPEs) have been used to sample distributions of many parameters, after which observations are used constrain the model to internally consistent and plausible values. These studies, which provide an efficient and comprehensive way to both evaluate and tune model parameters, use huge arrays of data representing complex changes in the model. While alternative methods to determine feature importance have been implemented in these studies, an approach such as the one presented in this work would provide an efficient way to interpret the effects of the parameter tuning. We further note recent developments in SHAP in which multiple targets can be predicted and evaluated, potentially providing a significant advantage for studies using PPEs.

In this work, we find that the SHAP values for opposite cloud phases do not balance out. Total liquid-phase values outweigh the total ice-phase values. This finding, along with the non-linearities and cloud-type dependencies of the system, suggests that changing our cloud parameterisations to simply move mass from one phase to another in order to balance the liquid and ice phases may not remove the SWCRETOA bias entirely. We suggest that a concerted effort is required to improve the individual representation of each phase in a more physical way, which can take into consideration different environmental conditions. We suggest that the parameterisations of CCN and INP are a good place to start as, unlike other changes to the model e.g. changing the freezing temperatures, detrainment temperatures, ice crystal shape or growth rates:, they are derived independently and do not simply change phase partitioning. We are not suggesting that improvements to microphysical representations are futile (in fact the opposite). We do, however, suggest that they may have a lesser impact or, in some cases, undesirable impacts if not done with strong physical backing.

have shown that empirically forcing a model's INP concentrations can result in significant improvements in super-cooled liquid water fraction. Such work has begun for the UM model family. have explored the importance of marine organics and dust to INP concentrations in the SO and their subsequent control on cloud reflectivity, but a more concerted effort is needed. Specifically, we need to ensure the chemical and aerosol pathways responsible for both INP and CCN are (a) accurate and (b) coupled to the cloud scheme satisfactorily (two-way coupling preferred). In ACCESS-AM2, analysis has shown that CCN concentrations are significantly underestimated in the SO (Fiddes et al., 2024), while INP concentrations are not resolved by the aerosol scheme but rather parameterised without consideration of the compositional environment. This is a key area for development, a process that has been started by ; however, it still requires considerable work with a stronger connection to the aerosol scheme. We hope that future work in this space can make use of the methods presented in this study to holistically quantify how their changes to the model affect the cloud radiative system.

To summarise, we have provided a new method for understanding model biases using a nudged climate simulation and machine learning. We hope that this method can be applied to other fields to gain new insights into the complexity and the drivers of modelling biases and how model perturbations may improve or worsen such biases. When considered as a whole, the SO SWCRETOA bias is shown to be complex, with large non-linearities, cloud-type and regional dependencies, and compensating errors. Our results suggest that the liquid phase of clouds is the most important contributor to the SWCRETOA bias, more so than the ice phase, and that different higher/thinner clouds often behave in an opposing manner to that of the lower/thicker cloud types, in some cases having compensating effects on the radiative bias. We propose that we need to address the biases in the liquid and ice phase of SO cloud properties individually (i.e. more physically) in order to reduce the SWCRETOA bias. We expect that the best way to do this is to ensure the nucleating particles (CCN and INP) are resolved by an aerosol scheme and fully coupled to the microphysics.