the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 04 Sep 2024
Cell tracking -based framework for assessing nowcasting model skill in reproducing growth and decay of convective rainfall
Abstract. The rapid temporal evolution of convective rainfall poses a challenge for rainfall nowcasting models. With the growing potential of machine learning models for precipitation nowcasting to produce realistic-looking nowcasts for long lead times, it is important to investigate whether the nowcasts also produce realistic development for convective rainfall. Common verification metrics traditionally used to validate nowcasting models are often dominated by large-scale stratiform rainfall, and averaging the metrics across entire precipitation fields obscures how accurately the models replicate individual convective cells, which makes it difficult to distinguish the model skill for the growth and decay of convective rainfall. In this study, we present a convective cell tracking-based framework to investigate how accurately nowcasting models reproduce the development of convective rainfall. The framework consists of first identifying and tracking the convective cells in the input observation rainfall fields, and then identifying and tracking the cells separately in the target observations and the nowcast rainfall fields by continuing the cell tracks identified in the observations. Features describing the cells and cell tracks, such as the cell volume rain rate and area, are then extracted. In addition to the errors in these feature values, the models’ skill in reproducing the existence of convective cells is estimated by calculating several contingency-table metrics, such as the Critical Success Index. The results allow the analysis of how accurately the models reproduce the growth and decay of convective rainfall and quantify the differences between the models, for example, due to differences in how the models smooth the nowcasts, i.e., blurring. The framework also allows differentiation of the results based on the initial conditions of the cell tracks, demonstrated here by separating the tracks into decaying or growing cell tracks based on the cell status when the nowcast is created. The framework is demonstrated using four open-source advection-based models: the advection nowcast, S-PROG, and LINDA implemented in the pysteps library, and L-CNN, with data from the Swiss radar network. The results indicate that the L-CNN model reproduced the existence of convective cells best among the models and had smaller errors in the cell volume rain rate than LINDA and S-PROG. LINDA had the smallest underestimation in the cell mean rain rate, whereas S-PROG significantly overestimated the cell volume rain rate and area because of blurring.
- Preprint
(5472 KB) - Metadata XML
-
Supplement
(1821 KB) - BibTeX
- EndNote
Status: final response (author comments only)
-
RC1: 'Comment on gmd-2024-99', Kun Zheng, 04 Oct 2024
The authors proposed a cell tracking-based framework for evaluating nowcasting models' ability to predict the growth and decay of convective rainfall. Using the cell tracking-based framework, the authors assess the forecasting ability of different models at the cell scale for local convective systems, rather than focusing on pixel-by-pixel verification. Four nowcasting models (the advection nowcast, S-PROG, LINDA, and L-CNN) are evaluated using Swiss radar data, with L-CNN showing the best performance in forecasting convective cells. However, there are certain issues that need to be addressed:
1. What specific results can be compared between the framework based on cell tracking and existing frameworks to demonstrate its superiority? While the framework assesses models at the cell scale, no comparison is provided with traditional pixel-scale methods. The results mainly focus on highlighting differences among the models but do not demonstrate the advantages of the proposed framework over conventional approaches. Including metrics like MAE, MSE, and SSIM would offer a clearer comparison between the two methods.
2.There is a lack of clear justification or references for many parameter settings in the framework, such as the minimum area threshold for convective cells. This should be addressed with more detailed explanations.
3. It is essential to add a final comprehensive evaluation metric in the framework. Independent evaluation algorithms for cell occurrence and development could hinder broader model applicability. A combined metric reflecting overall predictive capability should be introduced as a final output.
4.Although the significance of cell splitting and merging in convective rainfall evolution is acknowledged, no concrete evaluation methods are provided. The four models tested show limited performance in this aspect, making it difficult to determine if the framework effectively evaluates these processes.
5. Why L-CNN performs better than other models? Analyzing specific factors within the L-CNN model that contribute to this performance could provide insights for improving nowcasting models.
6. This framework has difficulty handling complex weather phenomena, especially when multiple weather cells interact in the same area. By only tracking the "most representative" cell and simplifying merge-split operations, the framework's versatility is limited. A more detailed explanation of the framework 's applicability is needed from the authors.
7.The paper is relatively long, with some redundant sections. Please revising the content in sections “2 Data and nowcasting models” and “3 Cell tracking-based verification framework” . It would be useful to reduce the details on the four models and remove repetitive or non-essential information.
Citation: https://doi.org/10.5194/gmd-2024-99-RC1 -
RC2: 'Comment on gmd-2024-99', Anonymous Referee #2, 11 Oct 2024
Thank you for inviting me to review the paper: “Cell tracking-based framework for assessing nowcasting model skill in reproducing growth and decay of convective rainfall” by Ritvanen et al.
The topic of convection is important, as these are often the strongest of storms. Their understanding for the current climate is required, and once verification is achieved of such knowledge, then we can start to project ahead to estimate their statistics for raised levels of atmospheric greenhouse gas concentrations. Furthermore, better assessment for now supports the development of improved weather forecasting systems. Hence this manuscript is useful, starting at the very basic but important level – can we understand short timescale evolution of such storms.
To appeal to the more general non-meteorological reader, it is important that the definitions are set out early. An immediate question is: “Why storm tracking” (which is about predicting into the future) when we are trying to nowcast? Very early on, in the Abstract even, state that “Nowcasting” includes prediction, at least for the short-term ahead. Indeed, also in the Abstract, give a typical timescale for that. Hours?
The wording of the Abstract could be tightened. In particular, it would help to make clearer that this is not a paper that develops a tracking methodology, but instead compares four existing tracking algorithms. This should be mentioned earlier in the Abstract, rather than “The framework consists….” (which again gives the impression this paper actually builds a tracking algorithm). Indeed, the title might be better-worded, something like “A comparison of…”.
There is some ambiguity about the naming of the “four open-source advection-based models” (Line 17). Does “the advection nowcast” have a name, and presumably this is model one? Then models two and three are S-PROG and LINDA. And the 4th models is L-CNN. Ideally there is a consistency of naming and order of model description throughout the entire paper to help the reader.
The statement of typical timescale would also be useful at the very beginning of the Introduction, i.e. the first sentence. Line 52 talks about lead times of one or two hours.
Reading further into the paper, then it becomes clear the authors regard their framework as more of a toolbox of statistical methods to assess storm tracking for short periods ahead. For instance, the caption to Figure 1 implies this. This is all fine, but again, please help the reader to make it clearer earlier in the manuscript what this terminology refers to. The paper is evaluating existing tracking methods.
A key criticism I do have is the presentation in Figure 1, and an opportunity is lost to really illustrate what the paper achieves. First of all, I like the top right part of the diagram that illustrates how observed data is a storm for, say, …. t-2, t-1, t-0. Then storms are observed at t+1, t+2, t+3, t+4… Correct? So this is the named “target cell tracks”. And then these are compared against the predicted storm features. These are the “Nowcast cell tracks”. Hence this part of the diagram makes very clear the approach taken. However, the size needs to be increased, not least so the times “t” and subscripts can be seen.
Once the top right part of the diagram is improved, then the left-hand column makes more sense. The split between “2” and “3” is identical to the split between “2” and “3”, top right. Can this be made clearer? Then, as a further enhancement to the diagram, show where the features link to the comparisons. In other words, is there a way to have arrows linking the left to the right? If not, then make it really clear that the “4-5” label on the right-hand side informs boxes “4” and “5” on the left-hand side. Correct?
The caption of Figure 1 could be made much more informative. This is important, because people often extract diagrams from paper .pdfs for use in powerpoint presentations. As far as possible, a diagram and caption should be self-contained.
Similarly, a little more time spent in Figure 2 would help. What does the outer bound box represent (i.e. is there something special about the domain 41.8N to 49.1N, etc. Is there a good reason why the domain of just one model is shown? (i.e. L-CNN). Please move the colourbar away from the diagram slightly. And please make the font of the tick labels, the colourbar labels and the colourbar title much bigger. Also make the “x” radar locations larger in the diagram.
Figure 3 could be made really informative, but the presentation is especially poor. I simply cannot see the tick labels on the colourbars, and the individual panel headings I can only just see. As I understand it, the top row is times t-4, t-3, t-2, t-1, t-0, which is radar data used to initialise the tracking algorithms. Then the second row is time t+1, t+2, t+3, t+4, t+5, and this is radar data we hope the tracking algorithms can emulate. Finally, the 3rd-6th rows are the performance of the different tracking algorithms for the same times t+1 tp t+5.
A further issue is that Figure 3 is at a large spatial scale, so to the eye, it is almost impossible to see the changes between times. In other words, almost every panel looks the same. Either zoom in on a particular region of change. Or alternatively, maybe present in colour the edges of change, e.g. blue for where advanced and red for where receded.
I will assume all the statistical analysis is robust and correct, but again, it is important to present the findings well. Figure 4 appears as a key diagram, as histograms of continuing, stopping and starting cells with advective storms. But is this purely data, or are the four models involved somehow? I’m guessing it is purely data, while Figure 5 shows, in a similar way, changes for the four models. I like the format of presentation, but again, the legend for Figure 5 is difficult to see.
This paper makes a useful contribution to understanding very short-term forecasting, and I am happy to re-review the manuscript. I can envisage the paper being published, but the authors really need to spend more time on the presentation. It is the underlying science that matters the most of course, but when presentation falls below a threshold, there is a real risk that ambiguities or misunderstandings will occur by readers.
Citation: https://doi.org/10.5194/gmd-2024-99-RC2 - AC1: 'Review reply on gmd-2024-99', Jenna Ritvanen, 27 Nov 2024
Data sets
Data for the manuscript the manuscript "Cell tracking -based frame- work for assessing nowcasting model skill in reproducing growth and decay of convective rainfall" by Ritvanen et al. J. Ritvanen et al. https://doi.org/10.57707/fmi-b2share.627e6133c2594dc3945d14fe0ef9c922
Results for the manuscript "Cell tracking -based framework for assessing now- casting model skill in reproducing growth and decay of convective rainfall" by Ritvanen et al. J. Ritvanen et al. https://doi.org/10.57707/fmi-b2share.6a0bf074134741deb6067d319da81ff8
Model code and software
Cell tracking -based verification framework for nowcasts J. Ritvanen https://doi.org/10.5281/zenodo.11240431
pysteps: T-DaTing algorithm with splits & merges D. Nerini et al. https://doi.org/10.5281/zenodo.11242613
Viewed
| HTML | XML | Total | Supplement | BibTeX | EndNote | |
|---|---|---|---|---|---|---|
| 262 | 57 | 152 | 471 | 24 | 7 | 8 |
- HTML: 262
- PDF: 57
- XML: 152
- Total: 471
- Supplement: 24
- BibTeX: 7
- EndNote: 8
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1