The paper presents how the JEDI framework and associated tools can be used to evaluate the atmospheric composition models. It is interesting to see that the JEDI framework can be used beyond data assimilation. This capability of JEDI would be of interest to atmospheric chemistry researchers. It allows using JEDI in both data assimilation and verification within the same atmospheric modeling systems. The text is concise and easy to follow. I recommend publishing this manuscript after addressing the following comments:

• My main concern regarding this evaluation framework is that the authors do not show how the uncertainties are accounted for in the atmospheric composition observations. These uncertainties are taken into account in the data assimilation, but not much in model verification. For example, there are significant uncertainties in the satellite AOD observations. How can these uncertainties be taken into account in the model evaluation? This aspect of the model-observation comparison is not considered here.
• How can these tools be used to conduct object based verification?
• How about aircraft observations? For example, the observations obtained during the field campaigns.
• 160: Does CRTM allow treatment of aerosols with non-spherical shapes?
• 230: The AirNow network is run by the states.
• 275: I assume the WRF-Chem simulations used the biomass burning emissions generated sometime later. Therefore, I would not say these are “WRF-Chem forecasts,” but rather retrospective simulations. Add a Table to show the model settings, e.g. resolution, physics and chemistry schemes that were used in the WRF-Chem simulations.
• It is remarkable that the WRF-Chem model has strong underestimation of the NO2 columns over NYC. What is causing this? The emission inventories or some other model errors. How does this bias affect O3 simulations over NYC?
• The RMSE statistics: Is this the centered RMSE? If not, I suggest changing that as the bias statistics are already provided.
• Figure 5. This map does not provide much information. It would be better to replace it with two plots to show the PM2.5 and O3 bias maps.
• MERRA-2 evaluation: As it is stated, MERRA-2 is based on GEOS-5, which assimilates the AERONET observations. Therefore, it is not clear what is the purpose of the MERRA-2 evaluation using the AERONET data. The presented AOD satellite verification is quite useful.
• Figure 7: Please add the regression lines (slope and intercept)
• References: For WRF/WRF-Chem this paper should be cited too: Powers, J. G., Klemp, J. B., Skamarock, W. C., Davis, C. A., Dudhia, J., Gill, D. O., et al. (2017). THE WEATHER RESEARCH AND FORECASTING MODEL Overview, System Efforts, and Future Directions. Bulletin of the American Meteorological Society, 98(8), 1717–1737.