Articles | Volume 17, issue 11
https://doi.org/10.5194/gmd-17-4579-2024
© Author(s) 2024. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/gmd-17-4579-2024
© Author(s) 2024. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Methods for assessment of models
| 
10 Jun 2024
Methods for assessment of models |
| 10 Jun 2024
| 10 Jun 2024
A general comprehensive evaluation method for cross-scale precipitation forecasts
Bing Zhang
Key Laboratory of Transportation Meteorology of China Meteorological Administration, Nanjing Joint Institute for Atmospheric Sciences, Nanjing 210041, China
Mingjian Zeng
CORRESPONDING AUTHOR
Key Laboratory of Transportation Meteorology of China Meteorological Administration, Nanjing Joint Institute for Atmospheric Sciences, Nanjing 210041, China
School of Atmospheric Sciences, Nanjing University, Nanjing 210023, China
Zhengkun Qin
School of Atmospheric Sciences, Nanjing University of Information Science and Technology, Nanjing 210044, China
Joint Center of Data Assimilation Research and Applications, Nanjing University of Information Science and Technology, Nanjing 210044, China
Couhua Liu
National Meteorological Centre, Beijing 100081, China
Wenru Shi
Key Laboratory of Transportation Meteorology of China Meteorological Administration, Nanjing Joint Institute for Atmospheric Sciences, Nanjing 210041, China
Key Laboratory of Transportation Meteorology of China Meteorological Administration, Nanjing Joint Institute for Atmospheric Sciences, Nanjing 210041, China
Kefeng Zhu
Key Laboratory of Transportation Meteorology of China Meteorological Administration, Nanjing Joint Institute for Atmospheric Sciences, Nanjing 210041, China
Chunlei Gu
School of Atmospheric Sciences, Nanjing University, Nanjing 210023, China
Jialing Zhou
Key Laboratory of Transportation Meteorology of China Meteorological Administration, Nanjing Joint Institute for Atmospheric Sciences, Nanjing 210041, China
Related authors
Xiaoyong Zhuge, Xiaolei Zou, Lu Yu, Xin Li, Mingjian Zeng, Yilun Chen, Bing Zhang, Bin Yao, Fei Tang, Fengjiao Chen, and Wanlin Kan
Earth Syst. Sci. Data, 16, 1747–1769, https://doi.org/10.5194/essd-16-1747-2024, https://doi.org/10.5194/essd-16-1747-2024, 2024
Short summary
Short summary
The Himawari-8/9 level-2 operational cloud product has a low spatial resolution and is available only during the daytime. To supplement this official dataset, a new dataset named the NJIAS Himawari-8/9 Cloud Feature Dataset (HCFD) was constructed. The NJIAS HCFD provides a comprehensive description of cloud features over the East Asia and west North Pacific regions for the years 2016–2022 by 30 retrieved cloud variables. The NJIAS HCFD has been demonstrated to outperform the official dataset.
Minghui Zhang, Yunjie Chen, Fan Yang, and Zhengkun Qin
EGUsphere, https://doi.org/10.5194/egusphere-2025-1980, https://doi.org/10.5194/egusphere-2025-1980, 2025
Short summary
Short summary
Considering the key role of high-resolution surface observation temperature data in the study of surface atmospheric temperature in ocean regions, we propose a new two-stage deep learning model. The model is used to fill ocean surface temperature data missing from satellite observations due to the orbital clearance of polar satellites.
Xiaoyong Zhuge, Xiaolei Zou, Lu Yu, Xin Li, Mingjian Zeng, Yilun Chen, Bing Zhang, Bin Yao, Fei Tang, Fengjiao Chen, and Wanlin Kan
Earth Syst. Sci. Data, 16, 1747–1769, https://doi.org/10.5194/essd-16-1747-2024, https://doi.org/10.5194/essd-16-1747-2024, 2024
Short summary
Short summary
The Himawari-8/9 level-2 operational cloud product has a low spatial resolution and is available only during the daytime. To supplement this official dataset, a new dataset named the NJIAS Himawari-8/9 Cloud Feature Dataset (HCFD) was constructed. The NJIAS HCFD provides a comprehensive description of cloud features over the East Asia and west North Pacific regions for the years 2016–2022 by 30 retrieved cloud variables. The NJIAS HCFD has been demonstrated to outperform the official dataset.
Tingfeng Wu, Boqiang Qin, Anning Huang, Yongwei Sheng, Shunxin Feng, and Céline Casenave
Geosci. Model Dev., 15, 745–769, https://doi.org/10.5194/gmd-15-745-2022, https://doi.org/10.5194/gmd-15-745-2022, 2022
Short summary
Short summary
Most hydrodynamic models were initially developed based in marine environments. They cannot be directly applied to large lakes. Based on field observations and numerical experiments of a large shallow lake, we developed a hydrodynamic model by adopting new schemes of wind stress, wind waves, and turbulence for large lakes. Our model can greatly improve the simulation of lake currents. This study will be a reminder to limnologists to prudently use ocean models to study lake hydrodynamics.
Tongwen Wu, Rucong Yu, Yixiong Lu, Weihua Jie, Yongjie Fang, Jie Zhang, Li Zhang, Xiaoge Xin, Laurent Li, Zaizhi Wang, Yiming Liu, Fang Zhang, Fanghua Wu, Min Chu, Jianglong Li, Weiping Li, Yanwu Zhang, Xueli Shi, Wenyan Zhou, Junchen Yao, Xiangwen Liu, He Zhao, Jinghui Yan, Min Wei, Wei Xue, Anning Huang, Yaocun Zhang, Yu Zhang, Qi Shu, and Aixue Hu
Geosci. Model Dev., 14, 2977–3006, https://doi.org/10.5194/gmd-14-2977-2021, https://doi.org/10.5194/gmd-14-2977-2021, 2021
Short summary
Short summary
This paper presents the high-resolution version of the Beijing Climate Center (BCC) Climate System Model, BCC-CSM2-HR, and describes its climate simulation performance including the atmospheric temperature and wind; precipitation; and the tropical climate phenomena such as TC, MJO, QBO, and ENSO. BCC-CSM2-HR is our model version contributing to the HighResMIP. We focused on its updates and differential characteristics from its predecessor, the medium-resolution version BCC-CSM2-MR.
Cited articles
Ahijevych, D., Gilleland, E., Brown, B. G., and Ebert, E. E.: Application of spatial verification methods to idealized and NWP-gridded precipitation forecasts, Weather Forecast., 24, 1485–1497, https://doi.org/10.1175/2009WAF2222298.1, 2009.
Bi, B., Dai, K., Wang, Y., Fu, J., Cao, Y., and Liu, C.: Advances in techniques of quantitative precipitation forecast, J. Appl. Meteorol. Sci., 27, 534–549, https://doi.org/10.11898/1001-7313.20160503, 2016.
Casati, B., Wilson, L. J., Stephenson, D. B., Nurmi, P., Ghelli, A., Pocernich, M., Damrath, U., Ebert, E. E., Brown, B. G., and Mason, S.: Forecast verification: current status and future directions, Meteorol. Appl., 15, 3–18, https://doi.org/10.1002/met.52, 2008.
Chen, F., Chen, J., Wei, Q., Li, J., Liu, C., Yang, D., Zhao, B., and Zhang, Z.: A new verification method for heavy rainfall forecast based on predictability II: Verification method and test, Acta. Meteorol. Sin., 77, 28–42, https://doi.org/10.11676/qxxb2019.003, 2019.
Chen, H., Li, P., and Zhao, Y.: A review and outlook of verification and evaluation of precipitation forecast at convection-permitting resolution, Adv. Meteorol. Sci. Technol., 11, 155–164, https://doi.org/10.3969/j.issn.2095-1973.2021.03.018, 2021.
Donaldson, R. J., Dyer, R. M., and Krauss, M. J.: An objective evaluator of techniques for predicting severe weather events, Ninth Conference on Severe Local Storms, Norman, Oklahoma, United States of America, American Meteorological Society [preprint], 321–326, 1975.
Doolittle, M. H.: The verification of predictions, B. Philos. Soc. Washington, 7, 122–127, 1885.
Doolittle, M. H.: Association ratios, Bulletin of the Philosophical Society of Washington, 10, 83–87, 94–96, 1888.
Doswell III, C. A., Davies-Jones, R., and Keller, D. L.: On summary measures of skill in rare event forecasting based on contingency tables, Weather Forecast., 5, 576–585, https://doi.org/10.1175/1520-0434(1990)005<0576:OSMOSI>2.0.CO;2, 1990.
Ebert, E. E.: Fuzzy verification of high-resolution gridded forecasts: a review and proposed framework, Meteorol. Appl., 15, 51–64, https://doi.org/10.1002/met.25, 2008.
Finley, J. P.: Tornado predictions, Am. Meteorol. J., 1, 85–88, 1884.
Flueck, J. A.: A study of some measures of forecast verification, in: 10th Conf. Probability and Statistics in Atmospheric Science, Edmonton, Alberta, Canada, American Meteorological Society [preprint], 69–73, 1987.
Gandin, L. S., and Murphy, A. H.: Equitable scores for categorical forecasts, Mon. Weather Rev., 120, 361–370, https://doi.org/10.1175/1520-0493(1992)120<0361:ESSFCF>2.0.CO;2, 1992.
Gilbert, G. K.: Finley's tornado predictions, Am. Meteorol. J., 1, 166–172, 1884.
Gilleland, E., Ahijevych, D., Brown, B. G., Casati, B., and Ebert, E. E.: Intercomparison of Spatial Forecast Verification Methods. Weather Forecast., 24, 1416–1430, https://doi.org/10.1175/2009WAF2222269.1, 2009.
Gofa, F., Boucouvala, D., Louka, P., and Flocas, H. A.: Spatial verification approaches as a tool to evaluate the performance of high resolution precipitation forecasts, Atmos. Res., 208, 78–87, https://doi.org/10.1016/j.atmosres.2017.09.021, 2018.
Han, F., Tang, W., Zhou, C., Sheng, J., and Zhang, X.: Improving a precipitation nowcasting algorithm based on the SWAN system and related application assessment, Acta Meteorol. Sin., 81, 304–315, https://doi.org/10.11676/qxxb2023.20220066, 2023.
Hanssen, A. W. and Kuipers, W. J. A.: On the relationship between the frequency of rain and various meteorological parameters, Mededeelingen en Verhandelingen, 81, 2–15, 1965.
Hao, C., Yu, B., Dai, Y., Zhi, X., And Zhang, Y.: Statistical downscaling research on spatio-temporal distributions of summer precipitation across the Beijing region, Meteorol. Mon., 49, 843–854, https://doi.org/10.7519/j.issn.1000-0526.2023.050201, 2023.
Heidke, P.: Berechnung des erfolges und der güte der windstärkevorhersagen im sturmwarnungdienst, Geografika Annaler, 8, 301–349, 1926.
Liu, C., Lin, J., Dai, K., Cao, Y., and Wei, Q.: An evaluation method suitable for precipitation forecasts and services, Torrential Rain and Disasters, 41, 712–719, https://doi.org/10.12406/byzh.2021-203, 2022.
Liu, C., Dai, K., Lin, J., Wei, Q., Li N., Wang, B., Tang B., Guo Y., Zhu W., Tang J., and Zeng X.: Design and implementation of whole process evaluation program library of weather forecast, Meteorol. Mon., 49, 351–364, https://doi.org/10.7519/j.issn.1000-0526.2022.050902, 2023.
Liu, J., Ren, C., Zhao, Z., Chen, C., Wang, Y., and Cai, K.: Comparative analysis on verification of heavy rainfall forecasts in different regional models, Meteorol. Mon., 48, 1292–1302, https://doi.org/10.7519/j.issn.1000-0526.2022.050502, 2022.
Marzban, C.: Scalar measures of performance in rare-event situations, Weather Forecast., 13, 753–763, https://doi.org/10.1175/1520-0434(1998)013<0753:SMOPIR>2.0.CO;2, 1998.
Mason, I. B.: Dependence of the critical success index on sample climate and threshold probability, Aust. Meteorol. Mag., 37, 75–81, 1989.
Mason, I. B.: Binary events, in: Forecast verification: a practitioner's guide in atmospheric science, edited by: Jolliffe IT and Stephenson DB, John Wiley and Sons, Chichester, United Kingdom, 37–76, 2003.
Murphy, A. H.: The Finley affair: a signal event in the history of forecast verification, Weather Forecast., 11, 3–20, https://doi.org/10.1175/1520-0434(1996)011<0003:TFAASE>2.0.CO;2, 1996.
Murphy, A. H. and Daan, H.: Forecast evaluation, in: Probability, statistics, and decision making in the atmospheric sciences, edited by: Murphy, A. H. and Katz, R. W., Westview Press, Boulder, Colorado, United States of America, 379–437, 1985.
Palmer, W. C. and Allen, R. A.: Note on the accuracy of forecasts concerning the rain problem, U.S. Weather Bureau, 1–4, 1949.
Peirce, C. S.: The numerical measure of the success of prediction, Science, 4, 453–454, 1884.
Roberts, N. M. and Lean, H. W.: Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events, Mon. Weather Rev., 136, 78–97, https://doi.org/10.1175/2007MWR2123.1, 2008.
Schaefer, J. T.: The critical success index as an indicator of warning skill, Weather Forecast., 5, 570–575, https://doi.org/10.1175/1520-0434(1990)005<0570:TCSIAA>2.0.CO;2, 1990.
Shi, W., Li, X., Zeng, M., Zhang, B., Wang, H., Zhu, K., and Zhuge, X.: Multi-model comparison and high-resolution regional model forecast analysis for the “7⋅20” Zhengzhou severe heavy rain, T. Atmos. Sci., 44, 688–702, https://doi.org/10.13878/j.cnki.dqkxxb.20210823001, 2021.
Stephenson, D. B., Casati, B., Ferro, C. A. T., and Wilson, C. A.: The extreme dependency score: A non-vanishing measure for forecasts of rare events, Meteorol. Appl., 15, 41–50, https://doi.org/10.1002/met.53, 2008.
Su, A., Lü, X., Cui, L., Li, Z., Xi, L., and Li, H.: The Basic Observational Analysis of “7.20” Extreme Rainstorm in Zhengzhou, Torrential Rain and Disasters, 40, 445–454, https://doi.org/10.3969/j.issn.1004-9045.2021.05.001, 2021.
Tang, W., Zhou, Q., Liu, X., Zhu, W., and Mao, X.: Analysis on verification of national severe convective weather categorical forecasts, Meteorol. Mon., 43, 67–76, https://doi.org/10.7519/j.issn.1000-0526.2017.01.007, 2017.
Wang, X., Yang, H., Cui, C., Li, C., Qi, H., Du, M., Wang, J., and Wang, X.: Analysis of unusual climatic characteristics of precipitation and four typical extreme weather processes in China in 2021, Torrential Rain and Disasters, 41, 489–500, https://doi.org/10.12406/byzh.2022-045, 2022.
Wei, Q., Li, W., Peng, S., Xue, F., Zhao, S., Zhang, J., and Qi, D.: Development and application of national verification system in CMA, J. Appl. Meteorol. Sci., 30, 245–256, https://doi.org/10.11898/1001-7313.20190211, 2019.
Weygandt, S. S., Loughe, A. F., Benjamin, S. G., and Mahoney, J. L.: Scale sensitivities in model precipitation skill scores during IHOP, in: 22nd Conference Severe Local Storms, Hyannis, Massachusetts, United States of America, American Meteorological Society, 4–8 October 2004.
Wilks, D. S.: Statistical Methods in the Atmospheric Sciences, 2nd edn., Academic Press, 630 pp., 2006.
Yang, D., Gao, X., and Zhang, W.: Research and improvement of a new rainstorm forecast accuracy verification scheme, in: Proceedings of the 34th Annual Meeting of the Chinese Meteorological Society, S1 Disaster Weather Monitoring, Analysis and Forecast, 549–559, 2017.
Yang, Y., Yin, J., Wang, D., Liu, Y., Lu, Y., Zhang, W., and Xu, S.: ABM-based emergency evacuation modelling during urban pluvial floods: A “7.20” pluvial flood event study in Zhengzhou, Henan Province, Sci. China Earth Sci., 66, 282–291, https://doi.org/10.1007/s11430-022-1015-6, 2023.
Yates, E., Anquetin, S., Ducrocq, V., Creutin, J.-D., Ricard, D., and Chancibault, K.: Point and areal validation of forecast precipitation fields, Meteorol. Appl., 13, 1–20, https://doi.org/10.1017/S1350482705001921, 2006.
Zepeda-Arce, J., Foufoula-Georgiou, E., and Droegemeier, K. K.: Space-time rainfall organization and its role in validating quantitative precipitation forecasts, J. Geophys. Res., 105, 10129–10146, https://doi.org/10.1029/1999JD901087, 2000.
Zhang, B., Zeng, M., Huang, A., Qin, Z., Liu, C., Shi, W., Li, X., Zhu, K., Gu, C., and Zhou, J.: Software and Data for “A general comprehensive evaluation method for cross-scale precipitation forecasts”, Zenodo [code and data set], https://doi.org/10.5281/zenodo.10951799, 2024.
Zhang, Y., Yu, H., Zhang, M., Yang, Y., and Meng, Z.: Uncertainties and error growth in forecasting the record-breaking rainfall in Zhengzhou, Henan on 19–20 July 2021, Sci. China Earth Sci., 65, 1903–1920, https://doi.org/10.1007/s11430-022-9991-4, 2022.
Zhao, B. and Zhang, B.: Application of neighborhood spatial verification method on precipitation evaluation, Torrential Rain and Disasters, 37, 1–7, https://doi.org/10.3969/j.issn.1004-9045.2018.01.001, 2018.
Zhong, Q., Sun, Z., Chen, H., Li, J., and Shen, L.: Multi model forecast biases of the diurnal variations of intense rainfall in the Beijing-Tianjin-Hebei region, Sci. China Earth Sci., 65, 1490–1509, https://doi.org/10.1007/s11430-021-9905-4, 2022.
Short summary
By directly analyzing the proximity of precipitation forecasts and observations, a precipitation accuracy score (PAS) method was constructed. This method does not utilize a traditional contingency-table-based classification verification; however, it can replace the threat score (TS), equitable threat score (ETS), and other skill score methods, and it can be used to calculate the accuracy of numerical models or quantitative precipitation forecasts.
By directly analyzing the proximity of precipitation forecasts and observations, a precipitation...
