Using radar observations to evaluate 3-D radar echo structure simulated by the Energy Exascale Earth System Model (E3SM) version 1

Wang, Jingyu; Fan, Jiwen; Houze Jr., Robert A.; Brodzik, Stella R.; Zhang, Kai; Zhang, Guang J.; Ma, Po-Lun

doi:https://doi.org/10.5194/gmd-14-719-2021

Articles | Volume 14, issue 2

https://doi.org/10.5194/gmd-14-719-2021

© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/gmd-14-719-2021

© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 14, issue 2

Methods for assessment of models

|

03 Feb 2021

Methods for assessment of models |

| 03 Feb 2021

Using radar observations to evaluate 3-D radar echo structure simulated by the Energy Exascale Earth System Model (E3SM) version 1

Jingyu Wang, Jiwen Fan, Robert A. Houze Jr., Stella R. Brodzik, Kai Zhang, Guang J. Zhang, and Po-Lun Ma

Download

Final revised paper (published on 03 Feb 2021)
Preprint (discussion started on 25 May 2020)

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

SC1: 'executive editor comment on gmd-2020-98', Astrid Kerkweg, 25 May 2020
- AC1: 'Response to the comments', Jiwen Fan, 13 Aug 2020
RC1: 'Reviewer commnts', Anonymous Referee #1, 22 Jun 2020
- AC2: 'Response to Reviewer #1', Jiwen Fan, 13 Aug 2020
RC2: 'Review', Alain Protat, 25 Jun 2020
- AC3: 'Response to Reviewer #2', Jiwen Fan, 13 Aug 2020

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

AR by Jiwen Fan on behalf of the Authors (13 Aug 2020) Author's response Manuscript

ED: Referee Nomination & Report Request started (31 Aug 2020) by Christina McCluskey

RR by Peter May (15 Sep 2020)

Suggestions for revision or reasons for rejection

I am broadly happy with the paper and responses, except for the following.

Line 48: “Our goal is to provide a comprehensive evaluation of both horizontal pattern and vertical structure of cloud and precipitation.” – How can you do this if you do not discriminate between convective, stratiform and large scale stratiform processes given the fundamentally different processes, heating and drying profiles. Radar simulations on such large grid sizes are already problematic, but given a key part of a convective paramaterisation is the fraction of the grid, surely the model and interpretation should could be improved. This is clearly beyond the current work, but should be acknowledged and planned for future work.

CFADS without considering this are limited in value in my opinion. Processing the radar data for convective and stratiform fractions in observations is straightforward (e.g. Steiner’s algorithm) and readily compared with convective fraction in cumulus paramaterisations and would provide a clear test for the models.
L154

“In EAMv1, 50 sub-columns are used for calculating the mean radar reflectivity for a model grid box. There are 625 pixels inside each 1° grid for NEXRAD data to provide a probability density function (PDF) of observed reflectivity within the box. F. “ – how can this be reliably done if you do not know what fraction is convective given radically different reflectivity profiles and magnitudes – even for similar rain rates?

L211 –“ the reflectivity below 4 km is consistent”, but in the CFAD’s the model has an odd double peak in the reflectivity and low numbers of low reflectivity compared with the observations. Some of this could be associated with the downscaling, but could also be due to the lack of proper classifications when downscaling the model reflectivity. I think these differences are significant and potentially important. I would also note identifying issues such as this is exactly the point of papers such as this.

Figure 6 showing the diurnal cycle looks very odd. In the model there is essentially none despite the well known biases in global models for initial deep convection too early in the day and the observations show two very narrow peaks that certainly are not consistent with previous observations such as the cited Carbone and Tuttle paper. As the authors say, it may represent issues in triggering convection in the model, but without information on convective fraction it is difficult to know. Note that Carbone and Tuttle showed that the timing of diurnal maxima was longitude dependent with propagating modes that would smear the diurnal cycle. However, what you have plotted, the reflectivity maxima in the model may not be expected to vary much as long as there is some precipitation given the profile is paramaterised from the existence of rainfall. Frankly, I do not know what to make of the peaks in reflectivity in the observations. These seem very narrow. On what spatial scale are these reflectivity maxima and how has this been averaged in time? If it is on the 4 km grid they may be too low, depending on how the temporal sampling is done. For the moment, I am not sure what this figure adds and would delete it but keep some of the text noting the diurnal variations (that are discussed in your previous paper? ).

Hide

RR by Anonymous Referee #3 (24 Sep 2020)

RR by Anonymous Referee #4 (29 Sep 2020)

Suggestions for revision or reasons for rejection

Review of “Using radar observations to evaluate 3d radar echo structure simulated by the global model E3SM Version 1” by Jingyu Wang, Jiwen Fan, Robert A. Houze Jr., Stella R. Brodzik, Kai Zhang, Guang J. Zhang, and Po-Lun Ma

Summary of paper:
The manuscript presents results of an evaluation of the E3SM model against NEXRAD radar observations for the summer periods during 2014-2016. The authors used the COSP forward simulator package to generate radar reflectivity values from their model cloud fields and averaged both the sub-grid COSP output and the NEXRAD observations to a 1-degree horizontal grid and 1-km vertical grid for like-with-like comparison. The model average reflectivity exceeds the observed value slightly at 2-km height, but at heights above 4 km the model generally does not produce enough cloud above the threshold reflectivity value. Sensitivity testing considering the convection and cumulus parameterisations does not improve this model bias.

Review summary:
This is generally a well-written paper with good quality figures. The evaluation of NWP models against 3D cloud and precipitation observations is of great importance and the present evaluation against NEXRAD is novel. The methodology is incomplete or insufficiently justified in places, which leads to serious concerns about the results. Nevertheless, these concerns might be overcome with appropriate clarifications or revisions and as such publication may be considered after major corrections.

Major comment 1:

The study is hindered by its original objective to evaluated the model against GPM and hence the implementation of the 13.6 GHz frequency in COSP. There are two issues at stake, namely (a) whether the comparison of 13.6 GHz simulated reflectivity against S-band (3-GHz) is appropriate and (b) whether the implementation has been done appropriately.

(a) The authors justify the 13.6 GHz versus 3 GHz comparison by citing their Wang et al. (2019b) study. While that is a nice paper, it is not a sufficiently comprehensive evaluation of the 13.6 GHz reflectivity against the 3 GHz reflectivity to convince the reader that these two are interchangeable. The key figure in that paper (Figure 2) uses normalisation, which removes any excess (or deficit) in cloud detection, which is of importance for this study. The normalisation within cloud, also performed in that Figure, masks the reduction in reflectivity values obtained with GPM (13.6 GHz) both due to attenuation and due to Mie scattering.

Beyond this general unease with the comparison, there are various studies that suggest a necessary conversion from Ku (13.6 GHz) to S band, with different equations used for ice and liquid phases. In particular, recent studies using the GPM radar to calibrate ground-based radars use such conversions:

Warren, R. A., A. Protat, S. T. Siems, H. A. Ramsay, V. Louf, M. J. Manton, and T. A. Kane, 2018: Calibrating Ground-Based Radars against TRMM and GPM. J. Atmos. Oceanic Technol., 35, 323–346, https://doi.org/10.1175/JTECH-D-17-0128.1.

(b) It is not obvious that the implementation of 13.6 GHz in COSP is straightforward. The authors state that the simulator automatically uses Rayleigh scattering, but that cannot be appropriate under all circumstances, particularly if the focus is on convection. Attenuation will not only be significant below 1-km altitude: convective towers can cause attenuation in the ice phase as well. Similarly, the large hydrometeors found aloft may lead to Mie scattering. That the Rayleigh scattering assumption is inappropriate for the GPM PR has long been established in the literature, e.g.:

L'Ecuyer, T. S., and G. L. Stephens, 2002: An Estimation-Based Precipitation Retrieval Algorithm for Attenuating Radars. J. Appl. Meteor., 41, 272–285, https://doi.org/10.1175/1520-0450(2002)041<0272:AEBPRA>2.0.CO;2.

Having said that, it is entirely possible that the 13.6 GHz is a red herring here. If Rayleigh scattering is assumed, and no Mie scattering is included for large particles, the COSP calculation might as well be considered as if it were a 3 GHz radar. In that case, it is worth checking the COSP calculations for whether the frequency/wavelength matters.

The authors have at least two options here. Either the authors provide corrected calculations following (for example) the papers above, for instance by applying such corrections to the COSP-simulated reflectivity. Alternatively, the authors develop a standalone forward simulator. If the latter, it is reasonable to assume Rayleigh reflectivity at 3 GHz (S-band) for comparison against the NEXRAD observations. Given the model microphysics assumptions (as listed in Table 1) it is relatively straightforward to calculate the Rayleigh reflectivity from the model ice and liquid water contents. This would be the most appropriate way to compare the model to the NEXRAD observations, but obviously requires some additional data processing, which may be difficult if the original cloud 3D fields were not included in the output.

Major comment 2:

The lack of sufficiently high radar reflectivity aloft is concerning and while this could be a model bias, it would be helpful for the reader to have more information regarding the COSP calculations. In particular, in Table 1 the authors specify the density of ice and the distribution width. Following Morrison and Gettelman (2008), the remaining size distribution parameters lambda and N0 should be calculated from the mixing ratios directly. The COSP calculation will require the (consant) density of ice and distribution width as well as the (variable) lambda and N0, unless COSP has the appropriate information to calculate lambda and N0 itself from the mixing ratio. If COSP is not provided with the correct information, a constant lambda and N0 may be assumed by COSP, leading to erroneous calculations.

On a related note, if the E3SM has been evaluated against CloudSat and/or CALIPSO, that would provide helpful context to include about its ability to produce high-level cloud.

Morrison, H., and A. Gettelman, 2008: A New Two-Moment Bulk Stratiform Cloud Microphysics Scheme in the Community Atmosphere Model, Version 3 (CAM3). Part I: Description and Numerical Tests. J. Climate, 21, 3642–3659, https://doi.org/10.1175/2008JCLI2105.1.

Major comment 3:

It is not clearly justified why the authors averaged their data to the 1-degree grid scale, when most of the information on the sub-grid scale is available to them. Averaging to 1-degree comes with its own problems (e.g. how to treat “cloud-free” regions) that may end up masking model deficiencies and it may have led to the disappearance of the characteristic CFAD shape in the NEXRAD analysis. Perhaps in Section 3.1, once the authors have performed their analysis using the sub-grid information, the authors could include some justification as to why the following analysis is done on the 1-degree averaged data.

Minor comments:

Line 52-53: It is important to acknowledge in the introduction that these “convective processes” are not resolved by the model and that some sub-grid representation is needed. The evaluation can then be performed either on coarsened observations (as this study does) or on the sub-grid sampled model, as COSP does. Please include such a clarification in the introduction.

Line 53-55: It is not obvious that these scattering effects are such an issue at S-band. Iguchi et al. (2018) consider the GPM-DPR which are smaller wavelength. At S-band, Rayleigh scattering could potentially be assumed which would make forward simulation much easier (easier than considering the sub-grid sampling for convection). Please rephrase this statement and consider studies using S-band radars specifically. [NB It should be noted that these lines are likely a result of the authors’ original intent to evaluate the model against GPM PR observations.]

Line 69-72 and Line 98-99: It should be made clear to the reader that there is no difference in microphysics parameters between convective and stratiform (referring to Table 1). Some further clarification is then required regarding the sub-grid partition. Presumably, the model diagnoses a convective cloud fraction and a stratiform cloud fraction, with their respective water contents. These water contents may differ and therefore could lead to different simulated radar reflectivity. This is important information to include, so perhaps in Line 69-72 explain the convective-stratiform partition and in Line 98-99 clarify the typical differences between convective and stratiform water contents (noting that the microphysics parameters are the same).

Line 106-108: The adoption of model-specific parameters is not unique and is a widely used approach when implementing COSP (or when developing their own forward simulator). Perhaps rephrase: “Following general usage of COSP, we modified the microphysics assumptions…”. The Swales et al. (2018) paper explicitly mentions the need to “maintain consistency between COSP1 and the host model.”

Section 3.1: As stated above, the “out of the box” configuration of COSP is not advised and general use should always assume the model parameters. As such, it is recommended to remove the left-hand panels in Figure 2, as well as the standalone Figure 3, and focus instead on the differences between the model and observations from the right-hand panels. More specifically, in Figure 2: (1) Why does the x-axis start at 14 dBZ, when an 8 dBZ minimum reflectivity is considered? (2) What are the units of density? Per 2 dB (i.e. 2 dB bins)? (3) Why show these PDFs normalised? It should be important to note the absence of “cloud” above 8 dBZ as well. The authors should include a separate Figure showing the fraction of occurrence of Z>8 dBZ with height to compare this between model and observations.

Section 3.2 and Figure 4: How is the mean calculated? In Section 2, we learn that for the instantaneous observation/simulated output, the mean is calculated in linear Z units (so that cloud-free areas are 0) and then converted to dBZ, with an 8 dBZ threshold. But how are values below 8 dBZ considered when calculating these long-term means? Or are the means (and standard deviation and 95th percentile) in-cloud only? In either case, it is useful to understand the occurrence of Z>8dBZ, so please add this to Table 2 and as a separate set of maps to complement Figure 4. The occurrence could help explain the difference in the mean, as the model could compensate for missing higher values by having a higher “cloud” occurrence.

Section 3.3 and Figure 5: Again, normalization occurs in-cloud, so information is lost on the frequency of occurrence of cloud more generally. Could the authors comment on the difference in characteristic shape of the CFAD between NEXRAD and the model? The model follows the well-known shape with a maximum occurrence at dBZ that increases from cloud top to freezing level, and then slowly decreases or stays constant below the freezing level. That shape can be reproduced with NEXRAD, but it seems to have disappeared in the authors’ analysis – is that solely due to the averaging to 1 degree? Perhaps the choice of Z for cloud-free regions is important here?

Line 219: “Above 11km, the model completely fails to simulate any reflectivity”. There is some nuance here, as the authors use the 8-dBZ threshold. So: “Above 11km, the model fails to generate average reflectivity above 8 dBZ.” Assuming that the authors have access to the data, it would be useful to report the typical reflectivity values that are generated by the model at these altitudes, even if below 8 dBZ.

Line 238-240 and Figure 6: It is unclear what is actually being considered here. Is column-maximum reflectivity the maximum in a column on the 1-degree grid? What is then the “radar reflectivity” at a given “local time” in Figure 6? Is this an average over the entire CONUS, but only for grid boxes with this value above 8 dBZ? Or is it the maximum over the entire CONUS? The way this is calculated might partly explain the signals that appear, so all this needs to be clarified in the text.

Section 3.4 and Figure 7: As above, a general understanding of frequency of occurrence of Z>8dBZ would be useful in addition to these (normalised) diagrams.

Line 301-305: This conclusion should be removed, as should be the related Figures and discussion, as it is widely established that the microphysics assumptions of the forward simulator should be consistent with those in the host model (e.g. Swales et al., 2018).

Hide

ED: Publish subject to minor revisions (review by editor) (02 Oct 2020) by Christina McCluskey

AR by Jiwen Fan on behalf of the Authors (30 Oct 2020) Author's response Manuscript

ED: Referee Nomination & Report Request started (25 Nov 2020) by Christina McCluskey

RR by Anonymous Referee #4 (27 Nov 2020)

Suggestions for revision or reasons for rejection

Review of “Using Radar Observations to Evaluate 3D Radar Echo Structure Simulated by the Global Model E3SM Version 1” by Jingyu Wang, Jiwen Fan, Robert A. Houze Jr, Stella R. Brodzik, Kai Zhang, Guang J. Zhang, and Po-Lun Ma

The authors have provided a detailed and clear response to the reviews and the additions to the text are mostly good. However, the newly revealed details of treatment of the ZM scheme and its condensate in COSP has raised a major concern that must be addressed. Beyond that and some minor corrections, the manuscript may be considered for publication.

Major comment:

In the response to reviewers, and now also in L76-78, the authors provide the following information on treatment of convective cloud in their study: “…whereas the convective cloud fraction is not parameterized in mass flux-based ZM scheme (assumed to be <<1 for typical GCM resolutions such as at 1-degree grid spacing or coarser), and is diagnosed from cloud mass flux for cloud radiation calculation, which is treated as a tunable parameter.”

Typical GCM evaluation studies considering precipitation combine both stratiform (or large-scale) precipitation as well as the output from the convection scheme (convective precipitation). Similarly, GCM evaluation studies considering clouds in terms of their radiative properties (e.g. outgoing longwave radiation) consider the grid-box mean OLR, that will be composed of both the convective and stratiform (or large-scale) radiative cloud. An evaluation study that does not take into account contributions from the convection scheme would vastly underestimate total precipitation and overestimate OLR from the GCM. The authors cite for instance the Xie et al. (2018) study which indeed considers convective contributions to precipitation and notes their importance.

Similarly, the assumption that convective fraction is << 1 does not mean its impact on the results in this paper should be ignored. For 3D radar reflectivity, consider a grid-box with 5-10% convective cloud. This cloud may reach high altitudes with significant amount of condensate, equivalent to Z=20 dBZ. That would lead to a grid-box mean greater than 0 dBZ and would show up in the authors’ results. This may very well be the case in the GCM representation of MCS, which is brought up as a key point in the conclusions in L355-356: “This pattern suggests that the model is not adequately representing the mesoscale convective systems that dominate warm season rainfall in that region.” Of course, in the interpretation of NEXRAD, no such convective elements are removed from the analysis, so the exclusion of convective condensate from the COSP analysis leads to a serious inconsistency in the evaluation.

In brief:

1) I assume that condensate from the ZM scheme is not included in the COSP calculations. If this assumption is incorrect, the text will need to be adjusted to explicitly state how ZM convective cloud condensate is treated.

2) If my assumption is correct, however, the authors need to make clear in the paper how in a future study they could incorporate the ZM condensate. Presumably, using the same diagnosed cloud fraction as done in the radiation scheme would be the way forward. They will also need to make clear in L76-78 the potential shortcomings in this study for excluding convective cloud condensate.

Note that numerous studies using the CloudSat simulator incorporate convective cloud somehow in their analysis, starting with one of the first such studies:

Bodas‐Salcedo, A., Webb, M. J., Brooks, M. E., Ringer, M. A., William, K. D., Milton, S. F., and Wilson, D. R. (2008), Evaluating cloud systems in the Met Office global forecast model using simulated CloudSat radar reflectivities, J. Geophys. Res., 113, D00A13, doi:10.1029/2007JD009620.

Note also that the ZM changes the authors have made will only affect their simulated reflectivities indirectly, i.e. by causing changes in the stratiform condensate. It may be that if COSP were allowed to simulate reflectivity from convective condensate, much bigger changes would be seen.

Minor comments:

L22-24: It is important to note in the abstract that this finding is in contrast with previous studies on cloud top height.

L105-107: “Note the COSP … radar simulation.” This is an incorrect representation of COSP and should be removed. Depending on the COSP configuration used, it does use the hydrometeor mixing ratios (or the precipitation fluxes).

L138-140: “Note the particle size… 13.6 GHz.” and also L145: “because the simulated particles are too small to cause Mie scattering at these radar frequencies.” This is an incorrect interpretation of GCM representation of cloud particle sizes and also is not supported by the Marchand et al. (2009) reference (which refers to effective radius, which is not relevant to this discussion). Most GCMs do assume a particle size distribution and thus implicitly contain particles of various sizes in their large-scale cloud and precipitation schemes. Such large particle sizes would therefore be considered by COSP. However, I do agree with the authors (thanks to their detailed response) that at 13.6 GHz such particles of similar size to the wavelength of 2.2 cm are going to be rare, particularly in the stratiform part of the cloud. The non-Rayleigh scattering regime would not be noticed until Z>40 dBZ, which is outside the regime shown in Figure 1. Please remove L138-140, which is incorrect, and rephrase L145 to finish (for example): “…COSP simulator in this study, because the simulated condensates are not large enough to lead to non-Rayleigh scattering, which is typically observed at Z>40 dBZ for Ku-band (Matrosov, 1992).”

Matrosov, S.Y., 1992. Radar reflectivity in snowfall. IEEE transactions on geoscience and remote sensing, 30(3), pp.454-461.

(The Matrosov reference Figure 2 shows reductions of 2-5 dB at the heavest ice particle concentrations. More appropriate and more recent references can be found by the authors if they prefer.)

L193-196: It may be appropriate to add a 4th example here, pending the response to the major comment on the ZM convective condensate. Perhaps this means moving L201-204 here.

L198: “Therefore, a higher-order consistency between the COSP and the host model is warranted in future studies.” This statement ought to be revisited in the conclusions and the abstract, as it is a major point of this study.

L352: Add “as advocated by the COSP community (Swales et al. 2018)”

Hide

ED: Publish subject to minor revisions (review by editor) (04 Dec 2020) by Christina McCluskey

AR by Jiwen Fan on behalf of the Authors (06 Dec 2020) Author's response Manuscript

ED: Publish as is (11 Dec 2020) by Christina McCluskey

Download

The requested paper has a corresponding corrigendum published. Please read the corrigendum first before downloading the article.

Article (4601 KB)
Full-text XML

Short summary

This paper presents an evaluation of the E3SM model against NEXRAD radar observations for the warm seasons during 2014–2016. The COSP forward simulator package is implemented in the model to generate radar reflectivity, and the NEXRAD observations are coarsened to the model resolution for comparison. The model severely underestimates the reflectivity above 4 km. Sensitivity tests on the parameters from cumulus parameterization and cloud microphysics do not improve this model bias.