Articles | Volume 16, issue 20
https://doi.org/10.5194/gmd-16-6029-2023
© Author(s) 2023. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/gmd-16-6029-2023
© Author(s) 2023. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Methods for assessment of models
| 
27 Oct 2023
Methods for assessment of models |
| 27 Oct 2023
| 27 Oct 2023
A standardized methodology for the validation of air quality forecast applications (F-MQO): lessons learnt from its application across Europe
Lina Vitali
National Agency for New Technologies, Energy and Sustainable Economic Development (ENEA), Department for Sustainability, Bologna, Italy
Kees Cuvelier
European Commission – Joint Research Centre (JRC), Ispra, Italy
retired
Antonio Piersanti
National Agency for New Technologies, Energy and Sustainable Economic Development (ENEA), Department for Sustainability, Bologna, Italy
Alexandra Monteiro
CESAM, Department of Environment, University of Aveiro, Aveiro, Portugal
Mario Adani
National Agency for New Technologies, Energy and Sustainable Economic Development (ENEA), Department for Sustainability, Bologna, Italy
Roberta Amorati
Regional Agency for Prevention, Environment and Energy (ARPAE) of the Emilia-Romagna region, Bologna, Italy
Agnieszka Bartocha
ATMOTERM, Opole, Poland
Alessandro D'Ausilio
Flemish Institute for Technological Research (VITO), Mol, Belgium
Paweł Durka
Institute of Environmental Protection (IEP) – National Research Institute, Warsaw, Poland
Carla Gama
CESAM, Department of Environment, University of Aveiro, Aveiro, Portugal
Giulia Giovannini
Regional Agency for Prevention, Environment and Energy (ARPAE) of the Emilia-Romagna region, Bologna, Italy
Stijn Janssen
Flemish Institute for Technological Research (VITO), Mol, Belgium
Tomasz Przybyła
ATMOTERM, Opole, Poland
Michele Stortini
Regional Agency for Prevention, Environment and Energy (ARPAE) of the Emilia-Romagna region, Bologna, Italy
Stijn Vranckx
Flemish Institute for Technological Research (VITO), Mol, Belgium
Philippe Thunis
CORRESPONDING AUTHOR
European Commission – Joint Research Centre (JRC), Ispra, Italy
Related authors
Augustin Colette, Gaëlle Collin, François Besson, Etienne Blot, Vincent Guidard, Frederik Meleux, Adrien Royer, Valentin Petiot, Claire Miller, Oihana Fermond, Alizé Jeant, Mario Adani, Joaquim Arteta, Anna Benedictow, Robert Bergström, Dene Bowdalo, Jorgen Brandt, Gino Briganti, Ana C. Carvalho, Jesper Heile Christensen, Florian Couvidat, Ilia D’Elia, Massimo D’Isidoro, Hugo Denier van der Gon, Gaël Descombes, Enza Di Tomaso, John Douros, Jeronimo Escribano, Henk Eskes, Hilde Fagerli, Yalda Fatahi, Johannes Flemming, Elmar Friese, Lise Frohn, Michael Gauss, Camilla Geels, Guido Guarnieri, Marc Guevara, Antoine Guion, Jonathan Guth, Risto Hänninen, Kaj Hansen, Ulas Im, Ruud Janssen, Marine Jeoffrion, Mathieu Joly, Luke Jones, Oriol Jorba, Evgeni Kadantsev, Michael Kahnert, Jacek W. Kaminski, Rostislav Kouznetsov, Richard Kranenburg, Jeroen Kuenen, Anne Caroline Lange, Joachim Langner, Victor Lannuque, Francesca Macchia, Astrid Manders, Mihaela Mircea, Agnes Nyiri, Miriam Olid, Carlos Pérez García-Pando, Yuliia Palamarchuk, Antonio Piersanti, Blandine Raux, Miha Razinger, Lennard Robertson, Arjo Segers, Martijn Schaap, Pilvi Siljamo, David Simpson, Mikhail Sofiev, Anders Stangel, Joanna Struzewska, Carles Tena, Renske Timmermans, Thanos Tsikerdekis, Svetlana Tsyro, Svyatoslav Tyuryakov, Anthony Ung, Andreas Uppstu, Alvaro Valdebenito, Peter van Velthoven, Lina Vitali, Zhuyun Ye, Vincent-Henri Peuch, and Laurence Rouïl
EGUsphere, https://doi.org/10.5194/egusphere-2024-3744, https://doi.org/10.5194/egusphere-2024-3744, 2024
Short summary
Short summary
The Copernicus Atmosphere Monitoring Service – Regional Production delivers daily forecasts, analyses, and reanalyses of air quality in Europe. The Service relies on a distributed modelling production by eleven leading European modelling teams following stringent requirements with an operational design which has no equivalent in the world. All the products are full, free, open and quality assured and disseminated with a high level of reliability.
Ilaria D'Elia, Gino Briganti, Lina Vitali, Antonio Piersanti, Gaia Righini, Massimo D'Isidoro, Andrea Cappelletti, Mihaela Mircea, Mario Adani, Gabriele Zanini, and Luisella Ciancarella
Atmos. Chem. Phys., 21, 10825–10849, https://doi.org/10.5194/acp-21-10825-2021, https://doi.org/10.5194/acp-21-10825-2021, 2021
Short summary
Short summary
We present an analysis of modelled trends of PM10, NO2, and O3 airborne concentrations over the Italian territory in 2003–2010. Our analysis shows a general downward simulated trend for all pollutants, with good agreement between observed and modelled values and the model widening both coverage and significance of air concentration trends. Due to the complex atmospheric dynamics, emission reductions do not always result in decreasing concentrations, especially for secondary pollutants.
Alexander de Meij, Cornelis Cuvelier, Philippe Thunis, and Enrico Pisoni
Geosci. Model Dev., 18, 4231–4245, https://doi.org/10.5194/gmd-18-4231-2025, https://doi.org/10.5194/gmd-18-4231-2025, 2025
Short summary
Short summary
We assess relevance and utility indicators by evaluating nine Copernicus Atmospheric Monitoring Service models in calculated air pollutant values. For NO2, the results highlight difficulties at traffic stations. For PM2.5 and PM10 the bias and winter–summer gradients reveal issues. O3 evaluation shows that seasonal gradients are useful. Overall, the indicators reveal model limitations, yet there is a need to reconsider the strictness of some indicators for certain pollutants.
Marc Guevara, Augustin Colette, Antoine Guion, Valentin Petiot, Mario Adani, Joaquim Arteta, Anna Benedictow, Robert Bergström, Andrea Bolignano, Paula Camps, Ana C. Carvalho, Jesper Heile Christensen, Florian Couvidat, Ilia D’Elia, Hugo Denier van der Gon, Gaël Descombes, John Douros, Hilde Fagerli, Yalda Fatahi, Elmar Friese, Lise Frohn, Michael Gauss, Camilla Geels, Risto Hänninen, Kaj Hansen, Oriol Jorba, Jacek W. Kaminski, Rostislav Kouznetsov, Richard Kranenburg, Jeroen Kuenen, Victor Lannuque, Frédérik Meleux, Agnes Nyíri, Yuliia Palamarchuk, Carlos Pérez García-Pando, Lennard Robertson, Felicita Russo, Arjo Segers, Mikhail Sofiev, Joanna Struzewska, Renske Timmermans, Andreas Uppstu, Alvaro Valdebenito, and Zhuyun Ye
EGUsphere, https://doi.org/10.5194/egusphere-2025-1287, https://doi.org/10.5194/egusphere-2025-1287, 2025
Short summary
Short summary
Air quality models require hourly emissions to accurately represent dispersion and physico-chemical processes in the atmosphere. Since emission inventories are typically provided at the annual level, emissions are downscaled to a refined temporal resolution using temporal profiles. This study quantifies the impact of using new anthropogenic temporal profiles on the performance of an European air quality multi-model ensemble. Overall, the findings indicate an improvement of the modelling results.
Augustin Colette, Gaëlle Collin, François Besson, Etienne Blot, Vincent Guidard, Frederik Meleux, Adrien Royer, Valentin Petiot, Claire Miller, Oihana Fermond, Alizé Jeant, Mario Adani, Joaquim Arteta, Anna Benedictow, Robert Bergström, Dene Bowdalo, Jorgen Brandt, Gino Briganti, Ana C. Carvalho, Jesper Heile Christensen, Florian Couvidat, Ilia D’Elia, Massimo D’Isidoro, Hugo Denier van der Gon, Gaël Descombes, Enza Di Tomaso, John Douros, Jeronimo Escribano, Henk Eskes, Hilde Fagerli, Yalda Fatahi, Johannes Flemming, Elmar Friese, Lise Frohn, Michael Gauss, Camilla Geels, Guido Guarnieri, Marc Guevara, Antoine Guion, Jonathan Guth, Risto Hänninen, Kaj Hansen, Ulas Im, Ruud Janssen, Marine Jeoffrion, Mathieu Joly, Luke Jones, Oriol Jorba, Evgeni Kadantsev, Michael Kahnert, Jacek W. Kaminski, Rostislav Kouznetsov, Richard Kranenburg, Jeroen Kuenen, Anne Caroline Lange, Joachim Langner, Victor Lannuque, Francesca Macchia, Astrid Manders, Mihaela Mircea, Agnes Nyiri, Miriam Olid, Carlos Pérez García-Pando, Yuliia Palamarchuk, Antonio Piersanti, Blandine Raux, Miha Razinger, Lennard Robertson, Arjo Segers, Martijn Schaap, Pilvi Siljamo, David Simpson, Mikhail Sofiev, Anders Stangel, Joanna Struzewska, Carles Tena, Renske Timmermans, Thanos Tsikerdekis, Svetlana Tsyro, Svyatoslav Tyuryakov, Anthony Ung, Andreas Uppstu, Alvaro Valdebenito, Peter van Velthoven, Lina Vitali, Zhuyun Ye, Vincent-Henri Peuch, and Laurence Rouïl
EGUsphere, https://doi.org/10.5194/egusphere-2024-3744, https://doi.org/10.5194/egusphere-2024-3744, 2024
Short summary
Short summary
The Copernicus Atmosphere Monitoring Service – Regional Production delivers daily forecasts, analyses, and reanalyses of air quality in Europe. The Service relies on a distributed modelling production by eleven leading European modelling teams following stringent requirements with an operational design which has no equivalent in the world. All the products are full, free, open and quality assured and disseminated with a high level of reliability.
Giorgio Veratti, Alessandro Bigi, Michele Stortini, Sergio Teggi, and Grazia Ghermandi
Atmos. Chem. Phys., 24, 10475–10512, https://doi.org/10.5194/acp-24-10475-2024, https://doi.org/10.5194/acp-24-10475-2024, 2024
Short summary
Short summary
In a study of two consecutive winter seasons, we used measurements and modelling tools to identify the levels and sources of black carbon pollution in a medium-sized urban area of the Po Valley, Italy. Our findings show that biomass burning and traffic-related emissions (especially from Euro 4 diesel cars) significantly contribute to BC concentrations. This research offers crucial insights for policymakers and urban planners aiming to improve air quality in cities.
Philippe Thunis, Jeroen Kuenen, Enrico Pisoni, Bertrand Bessagnet, Manjola Banja, Lech Gawuc, Karol Szymankiewicz, Diego Guizardi, Monica Crippa, Susana Lopez-Aparicio, Marc Guevara, Alexander De Meij, Sabine Schindlbacher, and Alain Clappier
Geosci. Model Dev., 17, 3631–3643, https://doi.org/10.5194/gmd-17-3631-2024, https://doi.org/10.5194/gmd-17-3631-2024, 2024
Short summary
Short summary
An ensemble emission inventory is created with the aim of monitoring the status and progress made with the development of EU-wide inventories. This emission ensemble serves as a common benchmark for the screening and allows for the comparison of more than two inventories at a time. Because the emission “truth” is unknown, the approach does not tell which inventory is the closest to reality, but it identifies inconsistencies that require special attention.
Eduardo Torre-Pascual, Gotzon Gangoiti, Ana Rodríguez-García, Estibaliz Sáez de Cámara, Joana Ferreira, Carla Gama, María Carmen Gómez, Iñaki Zuazo, Jose Antonio García, and Maite de Blas
Atmos. Chem. Phys., 24, 4305–4329, https://doi.org/10.5194/acp-24-4305-2024, https://doi.org/10.5194/acp-24-4305-2024, 2024
Short summary
Short summary
We present an analysis of an intense air pollution episode of tropospheric ozone (O3) along the Atlantic coast of the Iberian Peninsula, incorporating both measured and simulated parameters. Our study extends beyond surface-level factors to include altitude-related parameters. These episodes stem from upper-atmosphere O3 accumulation in preceding days, transported to surface layers, causing rapid O3 concentration increase.
Alexander de Meij, Cornelis Cuvelier, Philippe Thunis, Enrico Pisoni, and Bertrand Bessagnet
Geosci. Model Dev., 17, 587–606, https://doi.org/10.5194/gmd-17-587-2024, https://doi.org/10.5194/gmd-17-587-2024, 2024
Short summary
Short summary
In our study the robustness of the model responses to emission reductions in the EU is assessed when the emission data are changed. Our findings are particularly important to better understand the uncertainties associated to the emission inventories and how these uncertainties impact the level of accuracy of the resulting air quality modelling, which is a key for designing air quality plans. Also crucial is the choice of indicator to avoid misleading interpretations of the results.
Philippe Thunis, Alain Clappier, Enrico Pisoni, Bertrand Bessagnet, Jeroen Kuenen, Marc Guevara, and Susana Lopez-Aparicio
Geosci. Model Dev., 15, 5271–5286, https://doi.org/10.5194/gmd-15-5271-2022, https://doi.org/10.5194/gmd-15-5271-2022, 2022
Short summary
Short summary
In this work, we propose a screening method to improve the quality of emission inventories, which are responsible for large uncertainties in air-quality modeling. The first step of screening consists of keeping only emission contributions that are relevant enough. In a second step, the method identifies large differences that provide evidence of methodological divergence or errors. We used the approach to compare two versions of the CAMS-REG European-scale inventory over 150 European cities.
Svetlana Tsyro, Wenche Aas, Augustin Colette, Camilla Andersson, Bertrand Bessagnet, Giancarlo Ciarelli, Florian Couvidat, Kees Cuvelier, Astrid Manders, Kathleen Mar, Mihaela Mircea, Noelia Otero, Maria-Teresa Pay, Valentin Raffort, Yelva Roustan, Mark R. Theobald, Marta G. Vivanco, Hilde Fagerli, Peter Wind, Gino Briganti, Andrea Cappelletti, Massimo D'Isidoro, and Mario Adani
Atmos. Chem. Phys., 22, 7207–7257, https://doi.org/10.5194/acp-22-7207-2022, https://doi.org/10.5194/acp-22-7207-2022, 2022
Short summary
Short summary
Particulate matter (PM) air pollution causes adverse health effects. In Europe, the emissions caused by anthropogenic activities have been reduced in the last decades. To assess the efficiency of emission reductions in improving air quality, we have studied the evolution of PM pollution in Europe. Simulations with six air quality models and observational data indicate a decrease in PM concentrations by 10 % to 30 % across Europe from 2000 to 2010, which is mainly a result of emission reductions.
Philippe Thunis, Alain Clappier, Alexander de Meij, Enrico Pisoni, Bertrand Bessagnet, and Leonor Tarrason
Atmos. Chem. Phys., 21, 18195–18212, https://doi.org/10.5194/acp-21-18195-2021, https://doi.org/10.5194/acp-21-18195-2021, 2021
Short summary
Short summary
Air pollution's origin in cities is still a point of discussion, and approaches to assess the city's responsibility for its pollution are not harmonized and thus not comparable, resulting in sometimes contradicting interpretations. We show that methodological choices can easily lead to differences of a factor of 2 in terms of responsibility outcome and stress that methodological choices and assumptions most often lead to a systematic and important underestimation of the city's responsibility.
Ilaria D'Elia, Gino Briganti, Lina Vitali, Antonio Piersanti, Gaia Righini, Massimo D'Isidoro, Andrea Cappelletti, Mihaela Mircea, Mario Adani, Gabriele Zanini, and Luisella Ciancarella
Atmos. Chem. Phys., 21, 10825–10849, https://doi.org/10.5194/acp-21-10825-2021, https://doi.org/10.5194/acp-21-10825-2021, 2021
Short summary
Short summary
We present an analysis of modelled trends of PM10, NO2, and O3 airborne concentrations over the Italian territory in 2003–2010. Our analysis shows a general downward simulated trend for all pollutants, with good agreement between observed and modelled values and the model widening both coverage and significance of air concentration trends. Due to the complex atmospheric dynamics, emission reductions do not always result in decreasing concentrations, especially for secondary pollutants.
Philippe Thunis, Alain Clappier, Matthias Beekmann, Jean Philippe Putaud, Cornelis Cuvelier, Jessie Madrazo, and Alexander de Meij
Atmos. Chem. Phys., 21, 9309–9327, https://doi.org/10.5194/acp-21-9309-2021, https://doi.org/10.5194/acp-21-9309-2021, 2021
Short summary
Short summary
Modelling simulations are used to identify the most efficient emission reduction strategies to reduce PM2.5 concentration levels in northern Italy. Results show contrasting chemical regimes and important non-linearities during wintertime, with the striking result that PM2.5 levels may increase when NOx reductions are applied in NOx-rich areas – a process that may have contributed to the absence of significant PM2.5 decrease during the COVID-19 lockdowns in many European cities.
Jérôme Barré, Hervé Petetin, Augustin Colette, Marc Guevara, Vincent-Henri Peuch, Laurence Rouil, Richard Engelen, Antje Inness, Johannes Flemming, Carlos Pérez García-Pando, Dene Bowdalo, Frederik Meleux, Camilla Geels, Jesper H. Christensen, Michael Gauss, Anna Benedictow, Svetlana Tsyro, Elmar Friese, Joanna Struzewska, Jacek W. Kaminski, John Douros, Renske Timmermans, Lennart Robertson, Mario Adani, Oriol Jorba, Mathieu Joly, and Rostislav Kouznetsov
Atmos. Chem. Phys., 21, 7373–7394, https://doi.org/10.5194/acp-21-7373-2021, https://doi.org/10.5194/acp-21-7373-2021, 2021
Short summary
Short summary
This study provides a comprehensive assessment of air quality changes across the main European urban areas induced by the COVID-19 lockdown using satellite observations, surface site measurements, and the forecasting system from the Copernicus Atmospheric Monitoring Service (CAMS). We demonstrate the importance of accounting for weather and seasonal variability when calculating such estimates.
Bart Degraeuwe, Enrico Pisoni, and Philippe Thunis
Geosci. Model Dev., 13, 5725–5736, https://doi.org/10.5194/gmd-13-5725-2020, https://doi.org/10.5194/gmd-13-5725-2020, 2020
Short summary
Short summary
To make decisions on how to improve air quality, it is useful to identify the main sources of pollution for an area of interest. Often these sources of pollution are identified with complex models that, even if accurate, are time consuming and complex. In this work we use another approach, simplified models, to accomplish the same task. The results, computed with two different set of simplified models, show the main sources of pollution for selected cities, and the associated uncertainties.
Cited articles
Adani, M., Piersanti, A., Ciancarella, L., D'Isidoro, M., Villani, M. G., and Vitali, L.: Preliminary Tests on the Sensitivity of the FORAIR_IT Air Quality Forecasting System to Different Meteorological Drivers, Atmosphere, 11, 574, https://doi.org/10.3390/atmos11060574, 2020.
Adani, M., D'Isidoro, M., Mircea, M., Guarnieri, G., Vitali, L., D'Elia, I., Ciancarella, L., Gualtieri, M., Briganti, G., Cappelletti, A., Piersanti, A., Stracquadanio, M., Righini, G., Russo, F., Cremona, G., Villani, M. G., and Zanini, G.: Evaluation of air quality forecasting system FORAIR-IT over Europe and Italy at high resolution for year 2017, Atmos. Pollut. Res., 13, 101456, https://doi.org/10.1016/j.apr.2022.101456, 2022.
Agarwal, S., Sharma, S., R., S., Rahman, M. H., Vranckx, S., Maiheu, B., Blyth, L., Janssen, S., Gargava, P., Shukla, V. K., and Batra, S.: Air quality forecasting using artificial neural networks with real time dynamic error correction in highly polluted regions, Sci. Total Environ., 735, 139454, https://doi.org/10.1016/j.scitotenv.2020.139454, 2020.
Alfaro, S. C. and Gomes, L.: Modeling mineral aerosol production by wind erosion: Emission intensities and aerosol size distributions in source areas, J. Geophys. Res.-Atmos., 106, 18075–18084, https://doi.org/10.1029/2000JD900339, 2001.
Bai, L., Wang, J., Ma, X., and Lu, H.: Air Pollution Forecasts: An Overview, Int. J. Environ. Res. Publ. He., 15, 780, https://doi.org/10.3390/ijerph15040780, 2018.
Baklanov, A. and Zhang, Y.: Advances in air quality modeling and forecasting, Glob. Transit., 2, 261–270, https://doi.org/10.1016/j.glt.2020.11.001, 2020.
Baklanov, A., Schlünzen, K., Suppan, P., Baldasano, J., Brunner, D., Aksoyoglu, S., Carmichael, G., Douros, J., Flemming, J., Forkel, R., Galmarini, S., Gauss, M., Grell, G., Hirtl, M., Joffre, S., Jorba, O., Kaas, E., Kaasik, M., Kallos, G., Kong, X., Korsholm, U., Kurganskiy, A., Kushta, J., Lohmann, U., Mahura, A., Manders-Groot, A., Maurizi, A., Moussiopoulos, N., Rao, S. T., Savage, N., Seigneur, C., Sokhi, R. S., Solazzo, E., Solomos, S., Sørensen, B., Tsegas, G., Vignati, E., Vogel, B., and Zhang, Y.: Online coupled regional meteorology chemistry models in Europe: current status and prospects, Atmos. Chem. Phys., 14, 317–398, https://doi.org/10.5194/acp-14-317-2014, 2014.
Baldauf, M., Seifert, A., Förstner, J., Majewski, D., Raschendorfer, M., and Reinhardt, T.: Operational Convective-Scale Numerical Weather Prediction with the COSMO Model: Description and Sensitivities, Mon. Weather Rev., 139, 3887–3905, https://doi.org/10.1175/MWR-D-10-05013.1, 2011.
Borrego, C., Monteiro, A., Ferreira, J., Miranda, A. I., Costa, A. M., Carvalho, A. C., and Lopes, M.: Procedures for estimation of modelling uncertainty in air quality assessment, Environ. Int., 34, 613–620, https://doi.org/10.1016/j.envint.2007.12.005, 2008.
Boylan, J. W. and Russell, A. G.: PM and light extinction model performance metrics, goals, and criteria for three-dimensional air quality models, Atmos. Environ., 40, 4946–4959, https://doi.org/10.1016/j.atmosenv.2005.09.087, 2006.
Cabaneros, S. M., Calautit, J. K., and Hughes, B. R.: A review of artificial neural network models for ambient air pollution prediction, Environ. Model. Softw., 119, 285–304, https://doi.org/10.1016/j.envsoft.2019.06.014, 2019.
Carnevale, C., Finzi, G., Pederzoli, A., Pisoni, E., Thunis, P., Turrini, E., and Volta, M.: A methodology for the evaluation of re-analyzed PM10 concentration fields: a case study over the PO Valley, Air Qual. Atmos. Health, 8, 533–544, https://doi.org/10.1007/s11869-014-0307-2, 2015.
Chang, J. C. and Hanna, S. R.: Air quality model performance evaluation, Meteorol. Atmos. Phys., 87, 167–196, https://doi.org/10.1007/s00703-003-0070-7, 2004.
Chemel, C., Sokhi, R. S., Yu, Y., Hayman, G. D., Vincent, K. J., Dore, A. J., Tang, Y. S., Prain, H. D., and Fisher, B. E. A.: Evaluation of a CMAQ simulation at high resolution over the UK for the calendar year 2003, Atmos. Environ., 44, 2927–2939, https://doi.org/10.1016/j.atmosenv.2010.03.029, 2010.
D'Elia, I., Briganti, G., Vitali, L., Piersanti, A., Righini, G., D'Isidoro, M., Cappelletti, A., Mircea, M., Adani, M., Zanini, G., and Ciancarella, L.: Measured and modelled air quality trends in Italy over the period 2003–2010, Atmos. Chem. Phys., 21, 10825–10849, https://doi.org/10.5194/acp-21-10825-2021, 2021.
Dennis, R., Fox, T., Fuentes, M., Gilliland, A., Hanna, S., Hogrefe, C., Irwin, J., Rao, S. T., Scheffe, R., Schere, K., Steyn, D., and Venkatram, A.: A framework for evaluating regional-scale numerical photochemical modeling systems, Environ. Fluid Mech., 10, 471–489, https://doi.org/10.1007/s10652-009-9163-2, 2010.
Doms, G. and Baldauf, M.: A Description of the Non Hydrostatic Regional COSMO-Model. Part I: Dynamics and Numeric. User Guide Documentation, http://www.cosmo-model.org (last access: 20 October 2023), 2018.
Eder, B., Kang, D., Rao, S. T., Mathur, R., Yu, S., Otte, T., Schere, K., Wayland, R., Jackson, S., Davidson, P., McQueen, J., and Bridgers, G.: Using National Air Quality Forecast Guidance to Develop Local Air Quality Index Forecasts, B. Am. Meteorol. Soc., 91, 313–326, https://doi.org/10.1175/2009BAMS2734.1, 2010.
Emery, C., Liu, Z., Russell, A. G., Odman, M. T., Yarwood, G., and Kumar, N.: Recommendations on statistics and benchmarks to assess photochemical model performance, J. Air Waste Manag. Assoc., 67, 582–598, https://doi.org/10.1080/10962247.2016.1265027, 2017.
EPA: Guidelines for Developing an Air Quality (Ozone and PM2.5) Forecasting Program, EPA-456/R-03-002 June 2003, 2003.
European Union: Directive 2004/107/EC of the European Parliament and of the Council of 15 December 2004 relating to arsenic, cadmium, mercury, nickel and polycyclic aromatic hydrocarbons in ambient air, 2004.
European Union: Directive 2008/50/EC of the European Parliament and of the Council of 21 May 2008 on ambient air quality and cleaner air for Europe, OJ L, 152, 2008.
European Union: Proposal for a DIRECTIVE OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL on ambient air quality and cleaner air for Europe (recast), 2022.
Georgieva, E., Syrakov, D., Prodanova, M., Etropolska, I., and Slavov, K.: Evaluating the performance of WRF-CMAQ air quality modelling system in Bulgaria by means of the DELTA tool, Int. J. Environ. Pollut., 57, 272–284, https://doi.org/10.1504/IJEP.2015.074512, 2015.
Ginoux, P., Chin, M., Tegen, I., Prospero, J. M., Holben, B., Dubovik, O., and Lin, S.-J.: Sources and distributions of dust aerosols simulated with the GOCART model, J. Geophys. Res.-Atmos., 106, 20255–20273, https://doi.org/10.1029/2000JD000053, 2001.
Guenther, A., Karl, T., Harley, P., Wiedinmyer, C., Palmer, P. I., and Geron, C.: Estimates of global terrestrial isoprene emissions using MEGAN (Model of Emissions of Gases and Aerosols from Nature), Atmos. Chem. Phys., 6, 3181–3210, https://doi.org/10.5194/acp-6-3181-2006, 2006.
Hanna, S. R. and Chang, J.: Setting Acceptance Criteria for Air Quality Models, in: Air Pollution Modeling and its Application XXI, Dordrecht, 479–484, https://doi.org/10.1007/978-94-007-1359-8_80, 2012.
Hintze, J. L. and Nelson, R. D.: Violin Plots: A Box Plot-Density Trace Synergism, Am. Stat., 52, 181–184, https://doi.org/10.1080/00031305.1998.10480559, 1998.
Hooyberghs, J., Mensink, C., Dumont, G., Fierens, F., and Brasseur, O.: A neural network forecast for daily average PM10 concentrations in Belgium, Atmos. Environ., 39, 3279–3289, https://doi.org/10.1016/j.atmosenv.2005.01.050, 2005.
Janssen, S. and Thunis, P.: FAIRMODE Guidance Document on Modelling Quality Objectives and Benchmarking (version 3.3), EUR 31068 EN, Publications Office of the European Union, Luxembourg, 2022, JRC129254, ISBN 978-92-76-52425-0, https://doi.org/10.2760/41988, 2022.
Janssen, S., Dumont, G., Fierens, F., and Mensink, C.: Spatial interpolation of air pollution measurements using CORINE land cover data, Atmos. Environ., 42, 4884–4903, https://doi.org/10.1016/j.atmosenv.2008.02.043, 2008.
Kang, D., Eder, B. K., Stein, A. F., Grell, G. A., Peckham, S. E., and McHenry, J.: The New England Air Quality Forecasting Pilot Program: Development of an Evaluation Protocol and Performance Benchmark, J. Air Waste Manag. Assoc., 55, 1782–1796, https://doi.org/10.1080/10473289.2005.10464775, 2005.
Kingma, D. P. and Ba, J.: Adam: A Method for Stochastic Optimization, arXiv preprint arXiv:1412.6980, https://doi.org/10.48550/arXiv.1412.6980, 2014.
Knaff, J. A. and Landsea, C. W.: An El Niño–Southern Oscillation Climatology and Persistence (CLIPER) Forecasting Scheme, Weather Forecast., 12, 633–652, https://doi.org/10.1175/1520-0434(1997)012<0633:AENOSO>2.0.CO;2, 1997.
Kukkonen, J., Olsson, T., Schultz, D. M., Baklanov, A., Klein, T., Miranda, A. I., Monteiro, A., Hirtl, M., Tarvainen, V., Boy, M., Peuch, V.-H., Poupkou, A., Kioutsioukis, I., Finardi, S., Sofiev, M., Sokhi, R., Lehtinen, K. E. J., Karatzas, K., San José, R., Astitha, M., Kallos, G., Schaap, M., Reimer, E., Jakobs, H., and Eben, K.: A review of operational, regional-scale, chemical weather forecasting models in Europe, Atmos. Chem. Phys., 12, 1–87, https://doi.org/10.5194/acp-12-1-2012, 2012.
Kushta, J., Georgiou, G. K., Proestos, Y., Christoudias, T., Thunis, P., Savvides, C., Papadopoulos, C., and Lelieveld, J.: Evaluation of EU air quality standards through modeling and the FAIRMODE benchmarking methodology, Air Qual. Atmos. Health, 12, 73–86, https://doi.org/10.1007/s11869-018-0631-z, 2019.
Mailler, S., Menut, L., Khvorostyanov, D., Valari, M., Couvidat, F., Siour, G., Turquety, S., Briant, R., Tuccella, P., Bessagnet, B., Colette, A., Létinois, L., Markakis, K., and Meleux, F.: CHIMERE-2017: from urban to hemispheric chemistry-transport modeling, Geosci. Model Dev., 10, 2397–2423, https://doi.org/10.5194/gmd-10-2397-2017, 2017.
Marécal, V., Peuch, V.-H., Andersson, C., Andersson, S., Arteta, J., Beekmann, M., Benedictow, A., Bergström, R., Bessagnet, B., Cansado, A., Chéroux, F., Colette, A., Coman, A., Curier, R. L., Denier van der Gon, H. A. C., Drouin, A., Elbern, H., Emili, E., Engelen, R. J., Eskes, H. J., Foret, G., Friese, E., Gauss, M., Giannaros, C., Guth, J., Joly, M., Jaumouillé, E., Josse, B., Kadygrov, N., Kaiser, J. W., Krajsek, K., Kuenen, J., Kumar, U., Liora, N., Lopez, E., Malherbe, L., Martinez, I., Melas, D., Meleux, F., Menut, L., Moinat, P., Morales, T., Parmentier, J., Piacentini, A., Plu, M., Poupkou, A., Queguiner, S., Robertson, L., Rouïl, L., Schaap, M., Segers, A., Sofiev, M., Tarasson, L., Thomas, M., Timmermans, R., Valdebenito, Á., van Velthoven, P., van Versendaal, R., Vira, J., and Ung, A.: A regional air quality forecasting system over Europe: the MACC-II daily ensemble production, Geosci. Model Dev., 8, 2777–2813, https://doi.org/10.5194/gmd-8-2777-2015, 2015.
Marongiu, A., Angelino, E., Moretti, M., Malvestiti, G., and Fossati, G.: Atmospheric Emission Sources in the Po-Basin from the LIFE-IP PREPAIR Project, Open J. Air Pollut., 11, 70–83, https://doi.org/10.4236/ojap.2022.113006, 2022.
Masood, A. and Ahmad, K.: A review on emerging artificial intelligence (AI) techniques for air pollution forecasting: Fundamentals, application and performance, J. Clean. Prod., 322, 129072, https://doi.org/10.1016/j.jclepro.2021.129072, 2021.
Menut, L., Bessagnet, B., Khvorostyanov, D., Beekmann, M., Blond, N., Colette, A., Coll, I., Curci, G., Foret, G., Hodzic, A., Mailler, S., Meleux, F., Monge, J.-L., Pison, I., Siour, G., Turquety, S., Valari, M., Vautard, R., and Vivanco, M. G.: CHIMERE 2013: a model for regional atmospheric composition modelling, Geosci. Model Dev., 6, 981–1028, https://doi.org/10.5194/gmd-6-981-2013, 2013.
Mircea, M., Ciancarella, L., Briganti, G., Calori, G., Cappelletti, A., Cionni, I., Costa, M., Cremona, G., D'Isidoro, M., Finardi, S., Pace, G., Piersanti, A., Righini, G., Silibello, C., Vitali, L., and Zanini, G.: Assessment of the AMS-MINNI system capabilities to simulate air quality over Italy for the calendar year 2005, Atmos. Environ., 84, 178–188, https://doi.org/10.1016/j.atmosenv.2013.11.006, 2014.
Mittermaier, M. P.: The Potential Impact of Using Persistence as a Reference Forecast on Perceived Forecast Skill, Weather Forecast., 23, 1022–1031, https://doi.org/10.1175/2008WAF2007037.1, 2008.
Monteiro, A., Lopes, M., Miranda, A. I., Borrego, C., and Robert Vautard: Air pollution forecast in Portugal: a demand from the new air quality framework directive, Int. J. Environ. Pollut., 25, 4–15, https://doi.org/10.1504/IJEP.2005.007650, 2005.
Monteiro, A., Miranda, A. I., Borrego, C., and Vautard, R.: Air quality assessment for Portugal, Sci. Total Environ., 373, 22–31, https://doi.org/10.1016/j.scitotenv.2006.10.014, 2007a.
Monteiro, A., Miranda, A. I., Borrego, C., Vautard, R., Ferreira, J., and Perez, A. T.: Long-term assessment of particulate matter using CHIMERE model, Atmos. Environ., 41, 7726–7738, https://doi.org/10.1016/j.atmosenv.2007.06.008, 2007b.
Monteiro, A., Durka, P., Flandorfer, C., Georgieva, E., Guerreiro, C., Kushta, J., Malherbe, L., Maiheu, B., Miranda, A. I., Santos, G., Stocker, J., Trimpeneers, E., Tognet, F., Stortini, M., Wesseling, J., Janssen, S., and Thunis, P.: Strengths and weaknesses of the FAIRMODE benchmarking methodology for the evaluation of air quality models, Air Qual. Atmos. Health, 11, 373–383, https://doi.org/10.1007/s11869-018-0554-8, 2018.
Olesen, H. R.: Toward the Establishment of a Common Framework for Model Evaluation, in: Air Pollution Modeling and Its Application XI, edited by: Gryning, S.-E. and Schiermeier, F. A., Springer US, Boston, MA, 519–528, https://doi.org/10.1007/978-1-4615-5841-5_54, 1996.
Pernigotti, D., Gerboles, M., Belis, C. A., and Thunis, P.: Model quality objectives based on measurement uncertainty. Part II: NO2 and PM10, Atmos. Environ., 79, 869–878, https://doi.org/10.1016/j.atmosenv.2013.07.045, 2013.
Raffaelli, K., Deserti, M., Stortini, M., Amorati, R., Vasconi, M., and Giovannini, G.: Improving Air Quality in the Po Valley, Italy: Some Results by the LIFE-IP-PREPAIR Project, Atmosphere, 11, 429, https://doi.org/10.3390/atmos11040429, 2020.
Rahman, M. H., Agarwal, S., Sharma, S., Suresh, R., Kundu, S., Vranckx, S., Maiheu, B., Blyth, L., Janssen, S., Jorge, S., Gargava, P., Shukla, V. K., and Batra, S.: High-Resolution Mapping of Air Pollution in Delhi Using Detrended Kriging Model, Environ. Model. Assess., 28, 39–54, https://doi.org/10.1007/s10666-022-09842-5, 2023.
Russell, A. and Dennis, R.: NARSTO critical review of photochemical models and modeling, Atmos. Environ., 34, 2283–2324, https://doi.org/10.1016/S1352-2310(99)00468-9, 2000.
Ryan, W. F.: The air quality forecast rote: Recent changes and future challenges, J. Air Waste Manag. Assoc., 66, 576–596, https://doi.org/10.1080/10962247.2016.1151469, 2016.
Seigneur, C., Pun, B., Pai, P., Louis, J.-F., Solomon, P., Emery, C., Morris, R., Zahniser, M., Worsnop, D., Koutrakis, P., White, W., and Tombach, I.: Guidance for the Performance Evaluation of Three-Dimensional Air Quality Modeling Systems for Particulate Matter and Visibility, J. Air Waste Manag. Assoc., 50, 588–599, https://doi.org/10.1080/10473289.2000.10464036, 2000.
Skamarock, C., Klemp, B., Dudhia, J., Gill, O., Barker, D., Duda, G., Huang, X., Wang, W., and Powers, G.: A Description of the Advanced Research WRF Version 3. NCAR Tech. Note NCAR/TN-475+STR, https://doi.org/10.5065/D68S4MVH, 2008.
Sokhi, R. S., Moussiopoulos, N., Baklanov, A., Bartzis, J., Coll, I., Finardi, S., Friedrich, R., Geels, C., Grönholm, T., Halenka, T., Ketzel, M., Maragkidou, A., Matthias, V., Moldanova, J., Ntziachristos, L., Schäfer, K., Suppan, P., Tsegas, G., Carmichael, G., Franco, V., Hanna, S., Jalkanen, J.-P., Velders, G. J. M., and Kukkonen, J.: Advances in air quality research – current and emerging challenges, Atmos. Chem. Phys., 22, 4615–4703, https://doi.org/10.5194/acp-22-4615-2022, 2022.
Stortini, M., Arvani, B., and Deserti, M.: Operational Forecast and Daily Assessment of the Air Quality in Italy: A Copernicus-CAMS Downstream Service, Atmosphere, 11, 447, https://doi.org/10.3390/atmos11050447, 2020.
Szopa, S., Foret, G., Menut, L., and Cozic, A.: Impact of large scale circulation on European summer surface ozone and consequences for modelling forecast, Atmos. Environ., 43, 1189–1195, https://doi.org/10.1016/j.atmosenv.2008.10.039, 2009.
Tesche, T. W., Lurmann, F. R., Roth, P. M., Georgopoulos, P., and Seinfeld, J. H.: Improvement of procedures for evaluating photochemical models, Final report, Radian Corp., Sacramento, CA (USA), 1990.
Thunis, P. and Vitali, L.: Supporting data and tool, for the paper “A standardized methodology for the validation of air quality forecast applications (F-MQO): Lessons learnt from its application across Europe” (Version v2), Zenodo [data set], https://doi.org/10.5281/zenodo.7949868, 2023.
Thunis, P., Georgieva, E., and Pederzoli, A.: A tool to evaluate air quality model performances in regulatory applications, Environ. Model. Softw., 38, 220–230, https://doi.org/10.1016/j.envsoft.2012.06.005, 2012a.
Thunis, P., Pederzoli, A., and Pernigotti, D.: Performance criteria to evaluate air quality modeling applications, Atmos. Environ., 59, 476–482, https://doi.org/10.1016/j.atmosenv.2012.05.043, 2012b.
Thunis, P., Pernigotti, D., and Gerboles, M.: Model quality objectives based on measurement uncertainty. Part I: Ozone, Atmos. Environ., 79, 861–868, https://doi.org/10.1016/j.atmosenv.2013.05.018, 2013.
Zhang, B., Rong, Y., Yong, R., Qin, D., Li, M., Zou, G., and Pan, J.: Deep learning for air pollutant concentration prediction: A review, Atmos. Environ., 290, 119347, https://doi.org/10.1016/j.atmosenv.2022.119347, 2022.
Zhang, Y., Bocquet, M., Mallet, V., Seigneur, C., and Baklanov, A.: Real-time air quality forecasting, part I: History, techniques, and current status, Atmos. Environ., 60, 632–655, https://doi.org/10.1016/j.atmosenv.2012.06.031, 2012.
Short summary
Air quality forecasting models play a key role in fostering short-term measures aimed at reducing human exposure to air pollution. Together with this role comes the need for a thorough assessment of the model performances to build confidence in models’ capabilities, in particular when model applications support policymaking. In this paper, we propose an evaluation methodology and test it on several domains across Europe, highlighting its strengths and room for improvement.
Air quality forecasting models play a key role in fostering short-term measures aimed at...
