A comparison of the ability of the National Early Warning Score and the National Early Warning Score 2 to identify patients at risk of in-hospital mortality: A multi-centre database study

1Vital signs based aggregate early warning score (EWS) systems, which assign weights to each vital sign according to the deviation from assumed normal values, are recommended for routine use in UK hospitals,. In 2012, the Royal College of Physicians of London (RCPL) published a proposed National EWS (NEWS), which has now undergone extensive validation,,. In NEWS, oxygen saturations (SpO2) receive increasing weights for values of 95% or less, and oxygen therapy receives a flat weight. However, guidance for the management of patients with type II respiratory failure (T2RF),, and those deemed at risk of T2RF before blood gas analysis, suggests lower SpO2 values (88–92%) should be targeted. Consequently, it is suggested that the NEWS SpO2 weighting system is inappropriate for patients with/at risk of T2RF,,. Some authors suggest that this weighting risks inappropriate oxygen therapy for these patients, with potential deleterious consequences,.

2In December 2017, the RCPL published an update to NEWS - the National Early Warning Score 2 (NEWS2) - which includes several modifications to the NEWS vital sign weightings. To account for concerns about NEWS and T2RF, NEWS2 includes a new SpO2 scoring scale for patients with/at risk of T2RF. This scale, termed SpO2 scale 2 assigns weights at lower SpO2 thresholds than NEWS and combines these lower thresholds with weights for the use of supplemental oxygen at higher SpO2 levels, reflecting the concern of hyperoxia-induced hypercapnic respiratory failure (see appendix A1). Although the derivation of these thresholds is not presented, and NEWS2 is as yet unvalidated, NHS England has endorsed NEWS2′s use in acute and ambulance settings, and is considering the use of the Commissioning for Quality and Innovation (CQUIN) payment system, to encourage organisations to implement NEWS2 by March 2019.

3In this study, we used a large multi-centre dataset of vital signs to compare retrospectively the performance of NEWS2 and NEWS. We studied the performance of NEWS and NEWS2 in three risk groups: those with documented T2RF; those at risk of T2RF; and patients in neither of these groups.

4The database for this study was created with Health Research Authority (reference: 16/SC/0264 and 08/02/1394) approval. The study protocol is available online; we follow the TRIPOD statement for reporting.

5A database of vital sign observations was collated from adult (aged at least 16 years) acute admissions to the Oxford University Hospitals (OUH) group and Portsmouth Hospitals NHS Trust (PH) as part of the Hospital Alerting Via Electronic Noticeboard (HAVEN) dataset. Clinical staff recorded patients’ vital signs at the bedside using the System for Electronic Notification and Documentation (SEND, Drayson Health, www.draysonhealth.com) in OUH and VitalPAC™ (System C Healthcare, www.systemc.com) in PH. The following data were recorded: date and time of observation (automatically by SEND/VitalPAC™); heart rate, systolic blood pressure, respiratory rate, body temperature, neurological status using the Alert-Voice-Pain-Unresponsive (AVPU) scale, SpO2; and the patient’s inspired gas (air or supplemental oxygen) at the time of SpO2 measurement. The HAVEN database also contains administrative and patient demographic information, and information about the occurrence and timing of cardiac arrest, unanticipated intensive care unit (ICU) admission and hospital discharge status (dead/alive). Prescription data from the electronic patient record is also available within the database for OUH admissions.

6The study took place at five hospitals – the four hospitals in the OUH group [The John Radcliffe Hospital (large university hospital), The Horton General Hospital (small district general hospital), The Churchill hospital (large university cancer centre) and The Nuffield Orthopaedic Hospital] and a single large district general hospital, PH.

7All completed adult admissions to the four hospitals comprising the OUH group (January-December 2016) and to PH (January 2012 - December 2016) with at least one complete set of vital signs observations recorded electronically were considered. These study periods represent times of full deployment of electronic vital signs documentation in these hospitals. Patients discharged alive from the hospital before midnight on the day of admission and those with no vital signs recorded in the 24 h prior to discharge (as a proxy for patients on end-of-life pathways) were excluded from the analysis. For the main analysis, we combined admissions from all hospitals, but we also analysed data from each hospital trust separately (see appendix A3).

8The NEWS2 adjustment for patients with/at risk of T2RF differs from NEWS in the assignment of weights to measured SpO2 (NEWS weights SpO2 values below 96%; NEWS2 below 88%). Additionally, for patients with/at risk of T2RF, NEWS2 assigns weights for SpO2 values above 92% when receiving oxygen.

9The primary outcome was in-hospital death within 24 h of an observation set, in line with previous studies,. Secondary outcomes include cardiac arrest, unanticipated ICU admission, and either cardiac arrest, unanticipated ICU admission, or death within 24 h of an observation set. We present the results for all secondary outcomes, flagging those where insufficient outcomes exist (< 100), due to sample size, as recommended in the TRIPOD guidelines. All outcomes were obtained retrospectively from different clinical information systems, including the hospitals’ patient administration systems, the ICU clinical information systems, and the hospitals’ National Cardiac Arrest Audit (https://ncaa.icnarc.org) databases.

10Vital sign sets (see above) were recorded using SEND/VitalPAC™. Where the patient’s conscious level had been assessed only using the Glasgow Coma Scale (GCS), we converted GCS to an AVPU equivalent. Vital signs were then assigned weights for NEWS and NEWS2 scores (see appendix A1). The sum of the weights (aggregate score) results in the NEWS and NEWS2 value for each observation set. SEND (OUH) uses a modified EWS, CEWS, which assigns increasing weights to SpO2 values less than 94% and does not weight SpO2 values of 94% or above. Clinical staff entering vital signs data were therefore unaware of NEWS or NEWS2 scores. VitalPAC™ (PH) uses NEWS. To allow comparison with published analyses of NEWS,, and in line with previous vital-signs-based EWS research,,,, each vital sign set was analysed as independently associated with the outcome.

11For the analysis, we considered complete observation sets (i.e., sets with measurements of all vital signs), in line with previous NEWS studies,. The SEND system allows recording of incomplete vital sign sets, which is discouraged in the VitalPAC™ system. We did an a priori sub-analysis in which we used multiple imputation, a general-purpose and widely used approach to missing values which only occurred in the OUH dataset.

12Performance of NEWS and NEWS2 was assessed by discrimination using receiver operating characteristic (ROC) curve analysis (calibration was not assessed, as the EWS systems do not give estimates of absolute risk). We also assessed the effect of suggested thresholds for patient review (aggregate NEWS/NEWS2 scores of 5 or above, or 7 or above) by reporting sensitivity, specificity and positive predictive values. We also show SpO2 distributions for three different risk groups (see below). All analysis was performed using the R statistical software (v3.4.4) and ROC curves were calculated using the pROC package. Differences in the area under the ROC curve (AUROC), or c-statistic, between NEWS and NEWS2 were compared using bootstrapping (2000 samples). We did post-hoc sub-analyses of performance by institution (in light of the different patient numbers contributed). We also performed post-hoc efficiency curve analysis (as we were unable to conduct decision curve analysis as estimates of risk for a given score are not available).

13After exclusion criteria were applied, we categorised each admission according to the following risk groups:Patients with recorded T2RF, identified using the Adult Oxygen Prescription form of the current admission (OUH only).Patients at risk of T2RF, identified using the International Statistical Classification of Diseases and Related Health Problems-10 (ICD-10) classification for their concurrent or prior admission, with either of the following groups of diagnosis codes:J40-J44 (typically, 88% coded as J44) - patients with Chronic Obstructive Pulmonary Disease (COPD); orJ47 – patients with Bronchiectasis; orE84 – patients with Cystic Fibrosis; orE66 – patients with obesity and/or obesity hypoventilation syndrome; orG12, G47.3, G70-G71, M95.4, or Q67.8, with J96 – patients with respiratory failure (J96) and one of the following conditions: spinal muscular atrophy and other motor neuron disease (G12), sleep apnoea (G47.3), myasthenia gravis and other myoneural disorders (G70), muscular dystrophies (G71), acquired deformity of chest and rib (M95.4), or other congenital deformities of chest (Q67.8).Patients not at risk of T2RF, i.e., not in groups 1 or 2 above.We report the performance metrics of each scoring system for each of these risk groups. We report the results of the SpO2 scale 2 of NEWS2 in the third risk group (patients not at risk of T2RF) to demonstrate the effect of erroneous use of the scale in this population.

14Patients with recorded T2RF, identified using the Adult Oxygen Prescription form of the current admission (OUH only).

15Patients at risk of T2RF, identified using the International Statistical Classification of Diseases and Related Health Problems-10 (ICD-10) classification for their concurrent or prior admission, with either of the following groups of diagnosis codes:J40-J44 (typically, 88% coded as J44) - patients with Chronic Obstructive Pulmonary Disease (COPD); orJ47 – patients with Bronchiectasis; orE84 – patients with Cystic Fibrosis; orE66 – patients with obesity and/or obesity hypoventilation syndrome; orG12, G47.3, G70-G71, M95.4, or Q67.8, with J96 – patients with respiratory failure (J96) and one of the following conditions: spinal muscular atrophy and other motor neuron disease (G12), sleep apnoea (G47.3), myasthenia gravis and other myoneural disorders (G70), muscular dystrophies (G71), acquired deformity of chest and rib (M95.4), or other congenital deformities of chest (Q67.8).

16J40-J44 (typically, 88% coded as J44) - patients with Chronic Obstructive Pulmonary Disease (COPD); or

17J47 – patients with Bronchiectasis; or

18E84 – patients with Cystic Fibrosis; or

19E66 – patients with obesity and/or obesity hypoventilation syndrome; or

20G12, G47.3, G70-G71, M95.4, or Q67.8, with J96 – patients with respiratory failure (J96) and one of the following conditions: spinal muscular atrophy and other motor neuron disease (G12), sleep apnoea (G47.3), myasthenia gravis and other myoneural disorders (G70), muscular dystrophies (G71), acquired deformity of chest and rib (M95.4), or other congenital deformities of chest (Q67.8).

21Patients not at risk of T2RF, i.e., not in groups 1 or 2 above.

22We report the performance metrics of each scoring system for each of these risk groups. We report the results of the SpO2 scale 2 of NEWS2 in the third risk group (patients not at risk of T2RF) to demonstrate the effect of erroneous use of the scale in this population.

23NEWS was originally developed using a dataset with admissions to PH’s Medical Assessment Unit (MAU). The NEWS2 report does not identify a development dataset for NEWS2. The study evaluation database (HAVEN) includes data from all admissions to OUH and pH for the periods stated above. Vital sign data for all sites are present from hospital admission to hospital discharge/death. NEWS2 is recommended for use in all the included settings.

24A total of 251,266 distinct admissions were included. Fig. 1 shows the application of inclusion/exclusion criteria, resulting in the final cohort of admissions. All patients in the final dataset had at least one complete vital sign set. A total of 48,898 admissions were associated with patients at risk of T2RF, and 1394 with patients with documented T2RF (80.3% of whom also belong to the group of patients at risk of T2RF). Table 1 summarises the admission demographic descriptors and other clinical information for the three risk groups. Patients in risk groups 1 (documented T2RF) and 2 (at risk of T2RF) both had higher mortality rates (and rates of other adverse outcomes) when compared to patients who were not at risk (i.e. risk group 3).

25The distribution of SpO2 values for patients with documented T2RF is bell-shaped, whereas that for the group of patients who are not at risk was right skewed (Fig. 2). In patients with documented T2RF, 77.4% of admissions had at least one recorded SpO2 measurement above 92% on room air, compared with 98.7% in the non-risk group (Fig. 2).

26Performance metrics for the three risk groups for in-hospital death are presented in Table 2, and the corresponding ROC curves are represented in Fig. 3. Those for the secondary outcomes are shown in Table 3.

27Results of the sub-analyses by institution are shown in appendix A3. The effects of using multiple imputation to replace missing vital sign values are shown in appendix A4.

28In patients with documented T2RF, the AUROCs for predicting inpatient mortality within 24 h for the two scoring systems were as follows: NEWS 0.862 (95% CI: 0.848 to 0.875); NEWS2 0.841 (0.827 to 0.855) (Table 2). Using a threshold of 5 points, positive predictive values for NEWS and NEWS2 were 2.5% and 3.0% respectively. In patients at risk of T2RF, the AUROC for predicting inpatient mortality within 24 h for the two scoring systems were as follows: NEWS 0.881 (0.878 to 0.884); NEWS2 0.860 (0.857 to 0.864). Using a threshold of 5 points, positive predictive values for NEWS and NEWS2 were 3.2% and 2.7%, respectively.

29Our sub-analysis using multiple imputation to deal with missing values gave similar results (appendix A4).

30We calculated efficiency curves (see appendix A2) to compare the efficiency of NEWS and NEWS2. The curves demonstrate that, for the few patients with documented T2RF, the use of NEWS2 at the suggested RCPL cut-offs of 5 and 7 points reduces absolute staff workload by approximately 11% and 5% respectively, but at the expense of reduced sensitivity of approximately 10% and 14%, respectively. For patients at risk of T2RF, the use of NEWS2 at the suggested RCPL cut-offs of 5 and 7 points does not significantly decrease staff workload, but reduces sensitivity by 5–6%. Finally, if used in error for patients not at risk of T2RF at the suggested RCPL cut-offs, NEWS2 is slightly more sensitive than NEWS but, to achieve this, risks doubling the workload.

31This is the first study to evaluate the performance of NEWS2 in hospitalised patients who have documented T2RF or are at risk of it. For the primary outcome - in-hospital death within 24 h of an observation – NEWS2 demonstrated no improvement in discrimination over NEWS for patients with documented T2RF, but at the suggested RCPL cut-offs of 5 and 7 points, the positive predictive values (PPV) were higher for NEWS2 than NEWS. However, for patients at risk of T2RF, NEWS had superior discrimination and higher PPV compared to NEWS2. When applied to patients not at risk of T2RF (to simulate the impact of using NEWS2 in error in such patients) NEWS2 discriminated less well than NEWS and had lower PPV. Finally, NEWS2 did not improve discrimination for any of the secondary outcomes compared to NEWS.

32Modified scores have been suggested to account for chronically altered physiology in patients with respiratory-related conditions,,. One of these, CREWS, improved the positive predictive value compared to NEWS in patients with or at risk of T2RF (see appendix A5), but at the expense of decreasing sensitivity for events. However, such approaches challenge the premise that a universal EWS, with its attendant advantages, should be employed throughout hospitals. In NEWS2, assigning lower SpO2 thresholds together with heuristic weights for the use of supplemental oxygen at higher SpO2 values reflects the concern of hyperoxia-induced hypercapnic respiratory failure. However, encoding this concern as undertaken in NEWS2 does not improve discrimination in any of the three risk groups of admissions. Given the main purpose of EWS systems is to identify ill or deteriorating patients, the reduced sensitivity introduced by NEWS2 in patients with documented T2RF and those at risk of it is a disadvantage compared to NEWS. This reduced sensitivity could be ameliorated to an extent by reducing the trigger values for NEWS2, but this would increase staff workload, whilst also introducing further complexity.

33The performance of NEWS in this study is similar to that of the original derivation study for NEWS (AUROC, 0.89) supporting previous external evaluations of the scoring system, (see appendix A3 in Supplementary material, which describes the results considering admissions to each trust, separately).

34This study focuses on the patient groups for which the new SpO2 scoring “scale” in NEWS2 were intended. Robust electronic data capture allowed us to identify groups of patients admitted with/at risk of T2RF; this has not previously been undertaken. Unlike previous studies, our study includes vital signs taken throughout the patient’s hospital journey. The additional analyses, and the TRIPOD statement that guides our work further strengthen the findings of our study, promoting both clarity and interpretability.

35Our study relies on diagnostic codes and records of oxygen prescription to categorise patients with/at risk of T2RF, so patients could have been missed or misclassified. However, diagnostic coding for COPD has been shown to be relatively reliable, suggesting using this approach to identify those at risk of T2RF may also be reliable. In the case of oxygen prescriptions, the prescribing clinician’s assessment of whether or not the patient is a “carbon dioxide retainer” is recorded, and it seems likely that the same assessment would underlie the choice of SpO2 scale used. Our database does not include documentation of “new confusion”, which is now recommended to be part of the assessment of consciousness on for NEWS2; hence, we could not take account of this in our analysis. Nevertheless, as new confusion was not part of NEWS, our study clearly demonstrates the effect of the differences in oxygen SpO2 scales between the two systems for patients with T2RF. Moreover, the absence of this component is unlikely to have a different effect in the risk groups. By analysing each vital sign set as independently associated with outcome (allowing comparison with previous NEWS publications, ) we run the risk of over-representation of some patient groups. However, previous work suggests allowing an outcome to be represented only once has little effect on assessed outcomes. Evaluation of the secondary outcomes (cardiac arrest and unanticipated ICU admission) in the documented T2RF group should be interpreted with caution given the small number of outcomes (<100).

36We could find no performance benefit of NEWS2 in any diagnostic group compared to NEWS. If used in error in patients not at risk of T2RF, NEWS2 generally reduces discrimination compared to NEWS. Using NEWS2 instead of NEWS for patients with or at risk of T2RF reduces sensitivity for detecting patients with adverse outcomes. Improving sensitivity could be achieved by reducing the trigger values for NEWS2, but this would also increase staff workload.

37The recent endorsement by the RCPL and NHS England of the use of NEWS2 without underpinning evidence makes our study both important and urgent. Implementing NEWS2 requires additional staff training, and new multi-coloured charts, both of which are likely to be costly. The clinical impact of introducing NEWS2 may also have unexpected clinical consequences, some of which may also have financial impact.

38Applying the same “normal range” to patients with chronically abnormal physiology (e.g. COPD or heart failure) is a compelling criticism of using a single early warning score (EWS). It is certainly at odds with the interpretation of individual vital signs in clinical practice. However, this possible advantage needs to be counter-balanced with the simplicity of a single system. Applying different scores also creates a more complex protocol and observation chart, potentially increasing staff workload,. Ultimately, increasing score complexity has to be shown to improve performance for it to be worthwhile.

39Our study shows that the modifications made to NEWS2 (specifically, the alternative SpO2 scale), which increase chart complexity, are not likely to improve the detection of deterioration and/or reduce false alarms in patients with chronic respiratory disease.

40For patients at risk of, or with documented, T2RF, the changes proposed in NEWS2 do not improve the detection of adverse outcomes, including in-hospital death, unanticipated ICU admission, and cardiac arrest. The intent to account for known physiological differences in patients with chronic respiratory failure is laudable, as are the recommended improvements in the chart for recording oxygen prescriptions. However, the relationship between SpO2 values, oxygen therapy and the risk of adverse outcomes should be studied further before wide-scale adoption of NEWS2. In the interim, a more appropriate alternative to changing the weighting system for NEWS, might be to modify the clinical care escalation protocol and response to triggering.

41This publication presents independent research commissioned by the Health Innovation Challenge Fund (HICF-R9-524; WT-103703/Z/14/Z), a parallel funding partnership between the Department of Health and Wellcome Trust. The views expressed in this publication are those of the authors and not necessarily those of the Department of Health or Wellcome Trust. PJW is supported by the National Institute for Health Research (NIHR), Biomedical Research Centre, Oxford. SG is funded by a NIHR Doctoral Fellowship (DRF-2016-09-073). The views expressed are those of the authors and not necessarily those of the Funders.

42Study design: PJW, GSC, SG, JM, PES, GBS, DP; data preparation: OR, MAFP, DP; data analysis: OR, MAFP, DP, SG; data interpretation and writing up of the protocol and paper: all authors contributed.

43VitalPAC™, the system used to collect vital signs data in Portsmouth, is a collaborative development of The Learning Clinic Ltd (TLC) and Portsmouth Hospitals NHS Trust (PHT). At the time of the research, PHT had a royalty agreement with TLC to pay for the use of PHT intellectual property within the VitalPAC™ product. PS is employed by PHT. GS was an employee of PHT until 31/03/2011. DP was an employee of PHT until 31/07/2016. Until October 2015, PS and the wives of GS and DP were minority shareholders in TLC. GS is a member of the Royal College of Physicians of London’s National Early Warning Score (NEWS) Development and Implementation Group (NEWSDIG), which developed NEWS. DP assisted the Royal College of Physicians of London in the analysis of data validating NEWS. PW co-developed the System for Electronic Notification and Documentation (SEND), for which Drayson Health has purchased a sole licence. The company has a research agreement with the University of Oxford and royalty agreements with Oxford University Hospitals NHS Trust and the University of Oxford. Drayson Health may in the future pay PW personal fees.