Using wavelet transform and dynamic time warping to identify the limitations of the CNN model as an air quality forecasting system

Eslami, Ebrahim; Choi, Yunsoo; Lops, Yannic; Sayeed, Alqamah; Salman, Ahmed Khan

doi:https://doi.org/10.5194/gmd-13-6237-2020

Articles | Volume 13, issue 12

https://doi.org/10.5194/gmd-13-6237-2020

© Author(s) 2020. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/gmd-13-6237-2020

© Author(s) 2020. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 13, issue 12

Model evaluation paper

|

09 Dec 2020

Model evaluation paper |

| 09 Dec 2020

Using wavelet transform and dynamic time warping to identify the limitations of the CNN model as an air quality forecasting system

Ebrahim Eslami, Yunsoo Choi, Yannic Lops, Alqamah Sayeed, and Ahmed Khan Salman

Download

Final revised paper (published on 09 Dec 2020)
Supplement to the final revised paper
Preprint (discussion started on 09 Mar 2020)
Supplement to the preprint

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

RC1: 'Reviewer comments', Anonymous Referee #1, 25 Mar 2020
- AC1: 'response to the first reviewer', Yunsoo Choi, 03 Jun 2020
RC2: 'Important research on the limitations of using CNN in air quality forecasting', Anonymous Referee #2, 12 Apr 2020
- AC2: 'response to the second reviewer', Yunsoo Choi, 03 Jun 2020

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

AR by Anna Wenzel on behalf of the Authors (19 Jun 2020) Author's response

ED: Referee Nomination & Report Request started (20 Aug 2020) by Adrian Sandu

RR by Anonymous Referee #1 (07 Sep 2020)

RR by Anonymous Referee #3 (08 Sep 2020)

Suggestions for revision or reasons for rejection

‘Using wavelet transform and dynamic time warping to identify the limitations of the CNN model as an air quality forecasting system’ by Eslami et al.

Eslami et al. present a method based on wavelet transform and dynamic time warping (DTW) to characterize the quality of a machine-learning (ML) algorithm (convolutional neural network, CNN) for air quality forecasting (AQF). Using the example of two AQF applications, they show how wavelet transform and DTW can provide new insights into the strengths and weaknesses of the CNN model.
Better understanding the potential and limitations of ML algorithms for AQF applications is a topic that is rapidly gaining importance given the explosion of ML applications in this area. This paper makes a valuable contribution to this discussion by presenting a powerful analytical tool that can effectively highlight conditions under which the employed ML algorithm fails to produce satisfactory results. As such, the manuscript is highly suitable for publication in GMD. However, in its current form there are still some issues regarding the main message of the paper and how wavelet transform and DTW can be used to improve error characterization of ML applications.
For instance, the authors simultaneously say that the tested CNN models have ‘significant limitations’ and ‘show promising accuracy’, and generally seem to switch between the view that the ML model is either ‘bad’ or ‘good’. In reality, the CNN models – like chemical transport models – perform very well under some conditions and poorly under others. One of the powerful elements of the discussed statistical analysis tools is that they offer a method to identify these conditions and thus help the model developers better understand the strengths and limitations of the ML algorithms. This information also helps identify how the ML model might be improved, which is very powerful. The authors should stress this more clearly.
Another point that needs more discussion is the time dimension. The used CNN models seem to use snapshots of time-series data as inputs (rather than a window of the time-series) and are thus not designed to learn temporal relationships. This should be stated more clearly, as it means that the wavelet transform and DTW offer an assessment of a feature that is not directly optimized by the ML algorithm (which is a good thing).

Minor comments:
- Page 4, line 100: ‘general inability of the machine learning model’ seems a bit too harsh. I suggest to rephrase this.
- Page 5, line 124. Should be Figure 1, not Figure 3.
- Page 6, line 201: Please provide the definition of index of agreement
- Page 6, line 213: I’d be careful with the statement that NOx and VOC emissions are constant in time. These emissions have large diurnal and seasonal cycles.
- Page 7, line 251ff: maybe worth mentioning here the potential of long short-term memory (LSTM) algorithms to incorporate time dependency in the training?

Referee Report: PDF

Hide

ED: Publish subject to minor revisions (review by editor) (02 Oct 2020) by Adrian Sandu

AR by Lorena Grabowski on behalf of the Authors (13 Oct 2020) Author's response

ED: Publish as is (25 Oct 2020) by Adrian Sandu

Short summary

As using deep learning algorithms has become a popular data analytic technique, atmospheric scientists should have a balanced perception of their strengths and limitations so that they can provide a powerful analysis of complex data with well-established procedures. This study addresses significant limitations of an advanced deep learning algorithm, the convolutional neural network.