Evaluating epidemic forecasts in an interval format

Abstract

For practical reasons, many forecasts of case, hospitalization, and death counts in the context of the current Coronavirus Disease 2019 (COVID-19) pandemic are issued in the form of central predictive intervals at various levels. This is also the case for the forecasts collected in the COVID-19 Forecast Hub ( https://covid19forecasthub.org/ ). Forecast evaluation metrics like the logarithmic score, which has been applied in several infectious disease forecasting challenges, are then not available as they require full predictive distributions. This article provides an overview of how established methods for the evaluation of quantile and interval forecasts can be applied to epidemic forecasts in this format. Specifically, we discuss the computation and interpretation of the weighted interval score, which is a proper score that approximates the continuous ranked probability score. It can be interpreted as a generalization of the absolute error to probabilistic forecasts and allows for a decomposition into a measure of sharpness and penalties for over- and underprediction.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 38

1. 1.
   Collaborative efforts to forecast seasonal influenza in the United States, 2015–2016 · 2019

   No notice
2. 2.
   An open challenge to advance probabilistic forecasting for dengue epidemics · 2019

   Corrected
3. 3.

   Gneiting T, Raftery AE. Strictly proper scoring rules, prediction, and estimation. J Am Stat Assoc. 2007;102(477):359–378.

   Not in our library10.1198/016214506000001437

4. 4.

   UMass-Amherst Influenza Forecasting Center of Excellence. COVID-19 Forecast Hub; 2020. Accessible online at https://github.com/reichlab/covid19-forecast-hub. Last accessed 9 October 2020.

   Not in our library

5. 5.

   Ray EL, Wattanachit N, Niemi J, Kanji AH, House K, Cramer EY, et al COVID-19 Forecast Hub Consortium. Ensemble forecasts of coronavirus disease 2019 (COVID-19) in the U.S. Preprint, medRxiv. 2020

   Not in our library10.1101/2020.08.19.20177493

6. 6.

   Centers for Disease Control and Prevention. COVID-19 Forecasting: Background Information. Accessible online at https://www.cdc.gov/coronavirus/2019-ncov/cases-updates/forecasting.html. Last accessed 9 October 2020.

   Not in our library

7. 7.

   Centers for Disease Control and Prevention. Forecast the 2019–2020 Influenza Season Collaborative Challenge; 2019. Accessible online at https://predict.cdc.gov/api/v1/attachments/flusight_2019−2020/2019−2020_flusight_national_regional_guidance_final.docx.

   Not in our library

8. 8.

   Centers for Disease Control and Prevention. Forecast the 2018–2019 Influenza Season Collaborative Challenge; 2018. Accessible online at https://predict.cdc.gov/api/v1/attachments/flusight%202018%E2%80%932019/flu_challenge_2018-19_tentativefinal_9.18.18.docx

   Not in our library

9. 9.

   Reich NG, Brooks LC, Fox SJ, Kandula S, McGowan CJ, Moore E, et al A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States. Proc Natl Acad Sci. 2019;116(8):3146–3154.

   Not in our library10.1073/pnas.1812594116

10. 10.

    Bracher J. On the multibin logarithmic score used in the FluSight competitions. Proc Natl Acad Sci. 2019;116(42):20809–20810.

    Not in our library10.1073/pnas.1912147116

11. 11.

    Reich NG, Osthus D, Ray EL, Yamana TK, Biggerstaff M, Johansson MA, et al Reply to Bracher: Scoring probabilistic forecasts to maximize public health interpretability. Proc Natl Acad Sci. 2019;116(42):20811–20812.

    Not in our library10.1073/pnas.1912694116

12. 12.

    Gneiting T, Balabdaoui F, Raftery AE. Probabilistic forecasts, calibration and sharpness. J R Stat Soc B Stat Methodology. 2007;69(2):243–268.

    Not in our library10.1111/j.1467-9868.2007.00587.x

13. 13.

    Czado C, Gneiting T, Held L. Predictive model assessment for count data. Biometrics. 2009;65(4):1254–1261.

    Not in our library10.1111/j.1541-0420.2009.01191.x

14. 14.

    Kolassa S. Evaluating predictive count data distributions in retail sales forecasting. Int J Forecast. 2016;32(3):788–803.

    Not in our library10.1016/j.ijforecast.2019.02.017

15. 15.

    Held L, Meyer S, Bracher J. Probabilistic forecasting in infectious disease epidemiology: the 13th Armitage lecture. Stat Med. 2017;36(22):3443–3460.

    Not in our library10.1002/sim.7363

16. 16.

    Funk S, Camacho A, Kucharski AJ, Lowe R, Eggo RM, Edmunds WJ. Assessing the performance of real-time epidemic forecasts: A case study of Ebola in the Western Area region of Sierra Leone, 2014–15. PLoS Comput Biol. 2019;15(2):1–17.

    Not in our library10.1371/journal.pcbi.1006785

17. 17.

    Chowell G, Tariq A, Hyman JM. A novel sub-epidemic modeling framework for short-term forecasting epidemic waves. BMC Med. 2019;17(164).

    Not in our library10.1186/s12916-019-1406-6

18. 18.

    Chowell G, Rothenberg R, Roosa K, Tariq A, Hyman JM, Luo R. Sub-epidemic model forecasts for COVID-19 pandemic spread in the USA and European hotspots, February-May 2020. Preprint, medRxiv. 2020

    Not in our library10.1101/2020.07.03.20146159

19. 19.

    Laio F, Tamea S. Verification tools for probabilistic forecasts of continuous hydrological variables. Hydrol Earth Syst Sci Discuss. 2007;11(4):1267–1277.

    Not in our library10.5194/hess-11-1267-2007

20. 20.

    Gneiting T, Ranjan R. Comparing density forecasts using threshold- and quantile-weighted scoring rules. J Bus Econ Stat. 2011;29(3):411–422.

    Not in our library10.1198/jbes.2010.08110

21. 21.

    Hong T, Pinson P, Fan S, Zareipour H, Troccoli A, Hyndman RJ. Probabilistic energy forecasting: Global Energy Forecasting Competition 2014 and beyond. Int J Forecast. 2016;32(3):896–913.

    Not in our library10.1016/j.ijforecast.2016.02.001

22. 22.

    Diebold FX, Mariano RS. Comparing predictive accuracy. J Bus Econ Stat. 1995;13(3):253–263.

    Not in our library10.1080/07350015.1995.10524599

23. 23.

    Pinson P, Girard R. Evaluating the quality of scenarios of short-term wind power generation. Appl Energy. 2012;96:12–20.

    Not in our library10.1016/j.apenergy.2011.11.004

24. 24.

    Golestaneh F, Gooi HB, Pinson P. Generation and evaluation of space-time trajectories of photovoltaic power. Appl Energy. 2016;176:80–91.

    Not in our library10.1016/j.apenergy.2016.05.025

25. 25.

    Gneiting T, Stanberry LI, Grimit EP, Held L, Johnson NA. Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds. With discussion. Test. 2008;17:211–264.

    Not in our library10.1007/s11749-008-0114-x

26. 26.

    Juul JL, Græsbøll K, Christiansen LE, Lehmann S. Fixed-time descriptive statistics underestimate extremes of epidemic curve ensembles. Nat Phys. 2021; forthcoming.

    Not in our library10.1038/s41567-020-01121-y

27. 27.

    Winkler RL. Scoring rules and the evaluation of probabilities (with discussion). Test. 1996;5:1–60.

    Not in our library10.1007/bf02562681

28. 28.

    Bentzien S, Friederichs P. Decomposition and graphical portrayal of the quantile score. Q J Roy Meteorol Soc. 2014;140(683):1924–1934.

    Not in our library10.1002/qj.2284

29. 29.

    Ray EL, Sakrejda K, Lauer SA, Johansson MA, Reich NG. Infectious disease prediction with kernel conditional density estimation. Stat Med. 2017;36(30):4908–4929.

    Not in our library10.1002/sim.7488

30. 30.

    FluSight Network. Repository cdc-flusight-ensemble. 2020. Accessible online at https://github.com/FluSightNetwork/cdc-flusight-ensemble. Last accessed 13 October 2020.

    Not in our library

31. 31.

    Dawid AP. Statistical theory: The prequential approach. With discussion. J R Stat Soc A. 1984;147:278–292.

    Not in our library10.2307/2981683

32. 32.

    Gneiting T. Making and evaluating point forecasts. J Am Stat Assoc. 2011;106(494):746–762.

    Not in our library10.1198/jasa.2011.r10138

33. 33.

    Dunsmore IR. A Bayesian approach to calibration. J R Stat Soc B Methodol. 1968;30(2):396–405.

    Not in our library10.1111/j.2517-6161.1968.tb00740.x

34. 34.

    Winkler RL. A decision-theoretic approach to interval estimation. J Am Stat Assoc. 1972;67(337):187–191.

    Not in our library10.1080/01621459.1972.10481224

35. 35.

    Makridakis S, Spiliotis E, Assimakopoulos V. The M4 Competition: 100,000 time series and 61 forecasting methods. Int J Forecast. 2020;36(1):54–74.

    Not in our library10.1016/j.ijforecast.2019.04.014

36. 36.

    M Open Forecasting Center. The M5 Competition: Competitors’ Guide. 2020. Available from: https://mofc.unic.ac.cy/wp-content/uploads/2020/03/M5-Competitors-Guide-Final-10-March-2020.docx (downloaded 12 June 2020).

    Not in our library

37. 37.

    Askanazi R, Diebold FX, Schorfheide F, Shin M. On the comparison of interval forecasts. J Time Ser Anal. 2018;39(6):953–965.

    Not in our library10.1111/jtsa.12426

38. 38.

    Brehmer JR, Gneiting T. Scoring interval forecasts: Equal-tailed, shortest, and modal interval. Bernoulli. 2021; forthcoming. Preprint available from: https://arxiv.org/abs/2007.05709.

    Not in our library

Cited by papers we hold 2

• Gecko: A time-series model for COVID-19 hospital admission forecasting · 2022
  Not checked
• Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States · 2022
  Corrected

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.