Air quality forecasting models play a key role in fostering short-term measures aimed at reducing human exposure to air pollution. Together with this role comes the need for a thorough assessment of the model performances to build confidence in models’ capabilities, in particular when model applications support policymaking. In this paper, we propose an evaluation methodology and test it on several domains across Europe, highlighting its strengths and room for improvement.