Why the C-statistic is not informative to evaluate early warning scores and what metrics to use

Abstract

Metrics typically used to report the performance of an early warning score (EWS), such as the area under the receiver operator characteristic curve or C-statistic, are not useful for pre-implementation analyses. Because physiological deterioration has an extremely low prevalence of 0.02 per patient-day, these metrics can be misleading. We discuss the statistical reasoning behind this statement and present a novel alternative metric more adequate to operationalize an EWS. We suggest that pre-implementation evaluation of EWSs should include at least two metrics: sensitivity; and either the positive predictive value, number needed to evaluate, or estimated rate of alerts. We also argue the importance of reporting each individual cutoff value.

Shown for reading. Its publisher allows quoting it, not republishing it.

References 23

1. 1.

   Smith GB, Prytherch DR, Meredith P, Schmidt PE, Featherstone PI. The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation. 2013;84:465-70.

   Not in our library10.1016/j.resuscitation.2012.12.016

2. 2.

   Tirkkonen J, Olkkola KT, Huhtala H, Tenhunen J, Hoppu S. Medical emergency team activation: performance of conventional dichotomised criteria versus national early warning score. Acta Anaesth Scand. 2014;58:411-9.

   Not in our library10.1111/aas.12277

3. 3.

   Acutely ill patients in hospital: recognition of and response to acute illness in adults in hospital. National Institute for Health and Care Excellence. 2007. www.nice.org.uk/guidance/cg50. Accessed 02 Jul 2015.

   Not in our library

4. 4.

   National Early Warning Score (NEWS): standardising the assessment of acute-illness severity in the NHS—report of a working party. Royal College of Physicians of London. 2012. www.rcplondon.ac.uk/resources/national-early-warning-score-news. Accessed 02 Jul 2015.

   Not in our library

5. 5.

   D’Cruz R, Rubulotta F. Implementation of the National Early Warning Score in a teaching hospital [Abstract 0567]. Intensive Care Med. 2014;40 Suppl:160.

   Not in our library

6. 6.

   Gleeson L, Reynolds O, O’Connor P, Byrne D. Attitudes of doctors and nurses to the National Early Warning Score System. Irish J Med Sci. 2014;183 Suppl 4:193.

   Not in our library

7. 7.

   Jones M. NEWSDIG: The National Early Warning Score Development and Implementation Group. Clin Med. 2012;12:501-3.

   Not in our library10.7861/clinmedicine.12-6-501

8. 8.

   Romero-Brufau S, Huddleston JM, Naessens JM, Johnson MG, Hickman J, Morlan BW, Jensen JB, Caples SM, Elmer JL, Schmidt JA, et al. Widely used track and trigger scores: are they ready for automation in practice? Resuscitation. 2014;85:549-52.

   Not in our library10.1016/j.resuscitation.2013.12.017

9. 9.

   Escobar GJ, LaGuardia JC, Turk BJ, Ragins A, Kipnis P, Draper D. Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record. J Hosp Med. 2012;7:388-95.

   Not in our library10.1002/jhm.1929

10. 10.

    Donlon J, Levy H, Scriver C, Scriver CEA. Hyperphenylalaninemia: phenylalanine hydroxylase deficiency. The metabolic and molecular bases of inherited disease. 2004.

    Not in our library

11. 11.

    Dodd RY, Notari EP, Stramer SL. Current prevalence and incidence of infectious disease markers and estimated window-period risk in the American Red Cross blood donor population. Transfusion. 2002;42:975-9.

    Not in our library10.1046/j.1537-2995.2002.00174.x

12. 12.

    Burkhardt U, Mertens T, Eggers HJ. Comparison of two commercially available anti-HIV ELISAs: Abbott HTLV III EIA and Du Pont HTLV III-ELISA. J Med Virol. 1987;23:217-24.

    Not in our library10.1002/jmv.1890230303

13. 13.

    Stetler HC, Granade TC, Nunez CA, Meza R, Terrell S, Amador L, George JR. Field evaluation of rapid HIV serologic tests for screening and confirming HIV-1 infection in Honduras. Aids. 1997;11:369-75.

    Not in our library10.1097/00002030-199703110-00015

14. 14.

    McAlpine L, Gandhi J, Parry JV, Mortimer PP. Thirteen current anti-HIV-1/HIV-2 enzyme immunoassays: how accurate are they? J Med Virol. 1994;42:115-8.

    Not in our library10.1002/jmv.1890420203

15. 15.

    Smith GB, Prytherch DR, Schmidt PE, Featherstone PI. Review and performance evaluation of aggregate weighted “track and trigger” systems. Resuscitation. 2008;77:170-9.

    Not in our library10.1016/j.resuscitation.2007.12.004

16. 16.

    Smith GB, Prytherch DR, Schmidt PE, Featherstone PI, Higgins B. A review, and performance evaluation, of single-parameter “track and trigger” systems. Resuscitation. 2008;79:11-21.

    Not in our library10.1016/j.resuscitation.2008.05.004

17. 17.

    Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115:928-35.

    Not in our library10.1161/circulationaha.106.672402

18. 18.

    Graham KC, Cvach M. Monitor alarm fatigue: standardizing use of physiological monitors and decreasing nuisance alarms. Am J Crit Care. 2010;19:28-34.

    Not in our library10.4037/ajcc2010651

19. 19.

    Hannibal GB. Monitor alarms and alarm fatigue. AACN Adv Crit Care. 2011;22:418-20.

    Not in our library10.1097/nci.0b013e318232ed55

20. 20.

    Early warning systems: scorecards that save lives. Institute for Healthcare Improvement. http://www.ihi.org/resources/Pages/ImprovementStories/EarlyWarningSystemsScorecardsThatSaveLives.aspx. Accessed 02 Jul 2015.

    Not in our library

21. 21.

    Jund J, Rabilloud M, Wallon M, Ecochard R. Methods to estimate the optimal threshold for normally or log-normally distributed biological tests. Med Decis Making. 2005;25:406-15.

    Not in our library10.1177/0272989x05276855

22. 22.

    Hand DJ, Whitrow C, Adams NM, Juszczak P, Weston D. Performance criteria for plastic card fraud detection tools. J Oper Res Soc. 2008;59:956-62.

    Not in our library10.1057/palgrave.jors.2602418

23. 23.

    Srikureja W, Kyulo NL, Runyon BA, Hu KQ. MELD score is a better prognostic model than Child–Turcotte–Pugh score or discriminant function score in patients with alcoholic hepatitis. J Hepatol. 2005;42:700-6.

    Not in our library10.1016/j.jhep.2004.12.022

Cited by papers we hold 3

• Machine Learning–Based Early Warning Systems for Clinical Deterioration: Systematic Scoping Review · 2021
  Not checked
• Evaluation of Machine-Learning Algorithms for Predicting Opioid Overdose Risk Among Medicare Beneficiaries With Opioid Prescriptions · 2019
  No notice
• Scalable and accurate deep learning with electronic health records · 2018
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.