A New Risk Scheme to Predict Ischemic Stroke and Other Thromboembolism in Atrial Fibrillation: The ATRIA Study Stroke Risk Score

1Atrial fibrillation (AF) is a common arrhythmia that increases the risk for ischemic stroke by 4‐ to 5‐fold. Oral anticoagulant therapy is highly effective at reducing stroke risk in patients with AF but raises the risk of major bleeding.– Leading clinical practice guidelines– recommend a risk‐based approach to the anticoagulation decision in AF, usually based on the CHADS2 or CHA2DS2‐VASc stroke risk scores. However, these risk scores have only moderate ability to predict which patients will have a stroke.– To improve estimation of stroke risk, we developed and internally validated a new prediction model using the distinctively large experience of the Anticoagulation and Risk Factors in Atrial Fibrillation (ATRIA) AF cohort. We then externally validated the resulting new risk score in the recently assembled contemporary inception AF cohort, the ATRIA‐Cardiovascular Research Network (ATRIA‐CVRN).

2Assembly and validation of the ATRIA nonvalvular AF cohort have been described in detail previously. In brief, we used health plan databases to identify adult members of Kaiser Permanente Northern California who had an outpatient AF diagnosis between July 1, 1996, and December 31, 1997. We included all patients ≥18 years old with either 2 or more outpatient AF diagnoses (ICD‐9 code 427.31) or 1 outpatient AF diagnosis with ECG validation. The date of the first AF diagnosis was considered the patient's index date. Warfarin exposure was based on a validated algorithm using information from pharmacy and laboratory databases. Of the 13 559 patients in the ATRIA cohort, 9217 took warfarin at some point during follow‐up, contributing 33 497 person‐years of observation, and 10 927 contributed some time off warfarin during follow‐up. We used only person‐time off warfarin to develop our stroke risk model. In the Results section we analyze all person‐time off warfarin. The results comparing risk scores were very similar when we restricted the analysis to the 4342 patients who did not take warfarin at any point during follow‐up (data not shown).

3Clinical characteristics for patients in the ATRIA cohort were ascertained by searching inpatient, outpatient, laboratory, and pharmacy databases for the relevant ICD‐9 codes, medications, or lab values within the 5 years prior to the patient's index date (specific codes available by request) as well as administrative databases.– The Kaiser Permanente longitudinal diabetes registry was also used to identify patients with diabetes mellitus. These approaches for identifying comorbid conditions from electronic databases have been previously validated against a review of samples of patient medical records. Crude agreement ranged between 78% and 96% for individual risk factors. Methods for ascertaining proteinuria and estimated glomerular filtration rate (eGFR) are provided in a prior publication.

4ATRIA cohort members were followed from their index date through September 2003. Ascertainment of outcome events has been described previously. Follow‐up was censored at the date of the outcome event, death (ascertained via hospital databases, health plan reporting, Social Security Administration vital status files, and the California state death registry), or health plan disenrollment. Medical records of potential outcome events were reviewed by 2 physician reviewers. Ischemic stroke was defined as sudden onset of a neurologic deficit lasting >24 hours and not attributable to other identifiable causes. Other thromboembolic events were considered valid if they met the following criterion: sudden occlusion of an artery to a visceral organ or extremity documented by imaging, surgery, or pathology and not attributable to concomitant atherosclerosis or other etiology. Events first occurring in‐hospital or resulting from periprocedural complications were excluded. Each event was assigned a modified Rankin score indicating the level of disability at the time of hospital discharge.

5We externally validated the ATRIA score in the separate ATRIA‐CVRN AF cohort. The ATRIA‐CVRN study cohort is made up of 33 247 patients from Kaiser Permanente Northern California and also Kaiser Permanente Southern California aged 21 or older with incident atrial fibrillation (AF) or atrial flutter first diagnosed between January 2006 and June 2009 with confirmation by ECG or physician diagnosis in the electronic medical record. Validating diagnoses of AF included ≥1 inpatient diagnosis or ≥2 outpatient diagnoses. Unlike the ATRIA cohort, the ATRIA‐CVRN cohort did not exclude patients with mitral stenosis or a history of a valve replacement in the mitral or aortic positions; such patients account for 1.5% of the ATRIA‐CVRN cohort.

6Determination of baseline features, warfarin use, and outcome event occurrence and adjudication for the ATRIA‐CVRN cohort followed the same approach as for the ATRIA cohort except that for the ATRIA‐CVRN cohort emergency department visits that did not result in hospital admission could still count as valid outcome events. Patients were followed from their index date through June 2009. Follow‐up was censored at the date of the outcome event, death, or health plan disenrollment.

7We used a split‐sample approach to develop and internally validate the new stroke risk score. Patients with any periods off warfarin were randomly divided into a derivation cohort that accounted for approximately two thirds of person‐years contributed and a validation cohort that had the remaining one third of person‐years. We preselected 10 candidate predictor variables previously reported as stroke risk factors in atrial fibrillation: older age, female gender, prior ischemic stroke, diabetes, heart failure, hypertension, coronary artery disease (CAD), peripheral arterial disease (PAD), urine dipstick proteinuria, and low eGFR or end‐stage renal disease (ESRD) requiring dialysis.–,,– In addition, we considered total white blood cell count as an inflammatory marker and an episode of herpes zoster.– Age was categorized as <65, 65 to 74, 75 to 84, or ≥85 years old, and total white blood cell count was categorized as <8000, 8000 to 9999, or ≥10 000 per microliter. eGFR was dichotomized at ≥45 versus <45 mL/min per 1.73 m2 or ESRD. On the basis of univariate analysis results in the derivation cohort, we also tested an additional interaction term of age by prior stroke. All risk factor values were updated over the follow‐up period using the last value carried forward method. The follow‐up periods that did not have a preceding laboratory measurement going back as far as 5 years were considered normal (ie, WBC<8000/μL, eGFR ≥60 mL/min per 1.73 m2, and no proteinuria). These imputed normal values accounted for 3.5% of the person‐years for WBC, 2.8% of the person‐years for eGFR, and 22.2% of the person‐years for proteinuria.

8To minimize false‐positive variable selection, we constructed 1000 bootstrap samples based on the derivation two‐thirds cohort. For each sample, we used a time‐updated Cox proportional hazards model with the backward elimination method to determine predictors significant at the 0.05 level after variable selection. Variables consistently chosen in >60% of bootstrap samples were included in the final model. Traditional measures of model fit and their confidence intervals were calculated for both the derivation cohort and the validation cohort, including the c‐index for discrimination and the goodness‐of‐fit statistic for calibration. To generate a risk score, we assigned points to each variable proportional to its regression coefficients rounded to the nearest integer.

9We compared the discrimination capacity of the full‐range ATRIA stroke risk score with the full‐range CHADS2 and CHA2DS2‐VASc scores in the entire ATRIA off‐warfarin cohort using the c‐index. In addition, we collapsed the ATRIA stroke risk score into 3 categories on the basis of observed annual thromboembolic (TE) rates: low (TE rate <1.0%), moderate (TE rate 1.0 to <2.0%), and high (TE rate ≥2.0%). These rate cut points were based on a formal decision analysis. We compared the performance of the ATRIA 3‐category scheme to the CHADS2 and CHA2DS2‐VASc published 3‐category schemes both in terms of the c‐index and net reclassification improvement.–,

10We assessed the performance of our model in predicting severe TE events defined as those with a Rankin score ≥3 at hospital discharge (representing moderate to severe disability) or death within 30 days following the event. In addition, we tested the risk score in the primary prevention subset (ie, those individuals with no history of prior stroke).

11All model performance assessments were replicated in the separate ATRIA‐CVRN cohort. All analyses were conducted using SAS software, version 9.3 (SAS Institute, Inc, Cary, NC). The study was approved by the institutional review boards of the collaborating institutions. Waiver of informed consent was obtained because of the nature of the study.

12From the original ATRIA cohort of 13 559 AF patients, we accumulated 32 609 person‐years off warfarin contributed by 10 927 patients during the follow‐up period. The median time off warfarin was 2.4 years (range, <1 month to 7.2 years). We validated 685 TE events within this cohort (643 ischemic strokes and 42 other TE events) for an annualized rate of 2.1%. In the two‐thirds derivation cohort, there were 7284 patients with a total of 21 739 person‐years of follow‐up and 456 TE events. In the one‐third validation cohort, there were 3643 patients, 10 870 person‐years of follow‐up, and 229 TE events.

13In univariate analysis, all the selected predictor variables were significantly related to stroke risk except for a history of herpes zoster (Table 1). Thromboembolism risk increased monotonically across all 4 age categories, including a near‐doubling of risk for those ≥85 years compared with those 75 to 84 years old. The effect of age was particularly strong among the large group of patients without a prior stroke. However, the effect of older age was muted in patients with a history of prior stroke. Patients with a prior stroke in the 3 younger age categories all had high rates of subsequent stroke, about 6% per year. Patients ≥85 years with a history of stroke had the highest rate of stroke, >8% per year. This interaction of the effects of age and prior stroke was statistically significant (P<0.0001; Figure 1A).

14Eight variables were chosen in >60% of the 1000 bootstrap sample models: age (chosen for all 1000 bootstrap samples), prior stroke (for all 1000 samples), female (for 999 samples), diabetes mellitus (for 849 samples), heart failure (for 704 samples), hypertension (for 703 samples), proteinuria (for 870 samples), and eGFR<45 mL/min per 1.73 m2/ESRD (for 624 samples) plus the age by prior stroke interaction term (for 856 samples). Of note, CAD and PAD did not meet the bootstrap sample selection threshold and were not significant predictors when added to the final model (Table 2). In the multivariable model, the hazard ratio for CAD was 1.10 (P=0.27) and for PAD was 0.92 (P=0.70).

15The c‐index for the final model was 0.74 (95% CI, 0.72 to 0.76) for the derivation two thirds of the cohort, which decreased to 0.72 (95% CI, 0.68 to 0.75) when the model was tested in the one‐third validation cohort. The goodness‐of‐fit statistic did not show evidence of poor calibration in either cohort (P=0.14 and P=0.78 for the derivation and the validation sets, respectively). To construct a stroke risk score, we assigned points to predictor variables proportional to the size of their regression coefficients in the model (Table 2). Age and prior stroke were the dominant risk predictors. Possible scores ranged from 7 to 15 for patients with a prior stroke and from 0 to 12 for those with no history of stroke (Table 3).

16Thromboembolic event rates by the 3 scoring systems for the entire ATRIA cohort are shown in Table 4. For the full range of point scores, the c‐index was 0.73 (95% CI, 0.71 to 0.75) for the ATRIA score, 0.69 (95% CI, 0.67 to 0.71) for the CHADS2 score, and 0.70 (95% CI, 0.68 to 0.72) for the CHA2DS2‐VASc score (Table 5, column B).

17The ATRIA score was collapsed into low (0 to 5 points), moderate (6 points), and high (7 to 15 points) risk categories to fit annualized event rates of <1%, 1% to <2%, and ≥2% per year, respectively. The resulting c‐index was 0.69 (95% CI, 0.67, 0.71). By contrast, using the published low/moderate/high risk categories for the CHADS2 (0 to 1, 2 to 3, and 4 to 6 points, respectively) and the CHA2DS2‐VASc scores (0, 1, and ≥2 points, respectively), the resulting c‐indexes were 0.66 (95% CI, 0.64 to 0.68) and 0.58 (95% CI, 0.57 to 0.59), respectively (Table 5, column B). The low c‐index for the CHA2DS2‐VASc score reflected its very low threshold for its high‐risk category (Figure 2A).

18The ATRIA score accurately categorized large fractions of person‐years as low (47%) or high (40%) risk (Figure 2A, Table 4). By contrast, using published thresholds, the CHADS2 score placed most person‐years in low‐risk (50%) and moderate‐risk (45%) categories and the CHA2DS2‐VASc score placed most person‐years (81%) in the high‐risk category. The net reclassification improvement (NRI) from CHADS2 categories to the ATRIA categories was 26% (95% CI, 20% to 32%), composed predominantly of correct up‐classification from moderate risk (Table 6). The NRI from CHA2DS2‐VASc categories to the ATRIA categories was 27% (95% CI, 23% to 31%), resulting exclusively from down‐classification of CHA2DS2‐VASc high‐ and moderate‐risk categories.

19In the ATRIA‐CVRN cohort, 496 stroke or other thromboembolic events occurred (93.4% ischemic strokes) during the 26 263 person‐years of follow‐up off warfarin contributed by 25 306 patients, for a TE rate of 1.9% per year. The same pattern of age by prior stroke interaction was seen as in the ATRIA cohort (Figure 1B), although the TE rates in patients with a prior stroke and younger than 85 years were lower in the ATRIA‐CVRN cohort. All ATRIA score risk factors had strong significant univariate effects in ATRIA‐CVRN, although diabetes and eGFR were not significant in the multivariable model (data not shown). In ATRIA‐CVRN the c‐index was 0.70 (95% CI, 0.67 to 0.72) for the full‐range ATRIA score versus 0.66 (95% CI, 0.64 to 0.69) for CHADS2 and 0.68 (95% CI, 0.66 to 0.70) for CHA2DS2‐VASc (Table 5, column E). The distribution of low‐, moderate‐, and high‐risk categories were remarkably similar to that seen in the ATRIA cohort (Figure 2B). The c‐index deteriorated when the point scores were reduced to 3 categories, but the general pattern of performance seen in the ATRIA cohort persisted (Table 5, column E). The c‐index for the 3‐category CHA2DS2‐VASc score was again quite low, at 0.58 (95% CI, 0.58 to 0.59). The pattern of NRI values favoring the ATRIA score that was seen in the original cohort was reproduced in the ATRIA‐CVRN cohort (Table 6).

20Of the 685 TE events in the ATRIA data set, 399 were severe (Rankin score ≥3 at discharge or the patient died within 30 days after the event). When the full point scores were applied to this subset of severe events, the c‐index increased to 0.76 (95% CI, 0.74 to 0.79) for the ATRIA score, 0.72 (95% CI, 0.70 to 0.75) for CHADS2, and 0.73 (95% CI, 0.71 to 0.75) for CHA2DS2‐VASc (Table 5, column C). The NRI values for the 3‐category scores with outcomes restricted to severe events were similar to those for all outcome events (Table 6).

21We tested the 3 full‐range scores on the subset of 282 severe events in the ATRIA‐CVRN cohort. The c‐index for the ATRIA score was 0.75 (95% CI, 0.72 to 0.78), in excellent agreement with the derivation cohort result. By contrast, in the ATRIA‐CVRN cohort the c‐index deteriorated more for the CHADS2, 0.69 (95% CI, 0.66 to 0.72), and CHA2DS2‐VASc, 0.71 (95% CI, 0.68 to 0.74) scores (Table 5, column F). NRIs comparing the 3‐category ATRIA score with those of CHADS2 and CHA2DS2‐VASc were very similar to the values seen with the original ATRIA cohort (Table 6).

22Focusing only on cohort members without a history of prior stroke (primary prevention), the c‐index for the full‐range ATRIA score was 0.71 (95% CI, 0.69, 0.74) versus 0.66 (95% CI, 0.64, 0.69) for CHADS2 and 0.69 (95% CI, 0.67, 0.71) for the CHA2DS2‐VASc score (Table 5, column D). The same order was preserved in the ATRIA‐CVRN cohort with modest reductions in the c‐index (Table 5, column G). When the primary prevention analysis was restricted to severe outcomes, the c‐index improved (Table 5, column H and I). Agreement across the 2 cohorts was particularly good for the ATRIA score.

23We also tested point score cutoffs for the CHADS2 (0, 1, 2 to 6 points) and CHA2DS2‐VASc (0 to 2, 3, 4 to 9 points) scores that were optimized to fit the 1% and 2% per year thresholds for rate of thromboembolic (TE) events that we used in categorizing the ATRIA score (Table 4). Using these optimized cut points, the c‐index for the 3‐category CHA2DS2‐VASc score improved markedly to 0.68 (95% CI, 0.66 to 0.69) but was still lower than the c‐index for the ATRIA score. The c‐index for the CHADS2 score was largely unchanged, at 0.65 (95% CI, 0.64 to 0.67).

24Using these optimized cut points, the net reclassification improvement (NRI) for the ATRIA 3‐category score was 20% (95% CI, 17 to 24) versus the CHADS2 score and 8.6% (95% CI, 5.3 to 12) versus the CHA2DS2‐VASc score. These lower NRI results indicate less benefit from using the ATRIA score than had been observed with the published cut points. However, the ATRIA score still resulted in net correct reclassification compared with both the CHADS2 and the CHA2DS2‐VASc scores.

25Anticoagulants are highly effective in preventing ischemic stroke in persons with AF, but they can also cause major hemorrhage. Although prediction of risk of both ischemic stroke and major bleeding is relevant to the anticoagulation decision, formal decision analyses indicate that for most patients with AF, risk of ischemic stroke is the more important one.– Improved prediction of stroke risk in patients with AF would allow better targeted anticoagulant therapy. We used the large observational follow‐up of ATRIA AF cohort members not taking warfarin to optimize use of common clinical features to predict stroke risk. We validated the core risk factors used in the CHADS2 score but added features that we have previously reported to predict stroke (ie, female sex, renal dysfunction, and proteinuria)., Most importantly, we used a broader range of age categories, a decision consistent with multiple prior reports., We found strong amplification of stroke risk across the entire age range, with individuals aged ≥85 years at nearly double the risk of those aged 75 to 84 years. However, individuals who had had a stroke were at elevated risk regardless of age. Age, prior stroke, and their interaction were the dominant risk factors in our model. As with multiple prior studies, we did not observe significant incremental risk prediction from coronary disease, the major component of “VASc” in the CHA2DS2‐VASc score.–, Our resulting ATRIA risk scheme performed better among ATRIA cohort members than did the CHADS2 and CHA2DS2‐VASc schemes, which are recommended in leading clinical guidelines.– The c‐index was greater, and there was positive net reclassification improvement. From a practical point of view, more patients were accurately classified as low or high risk. Indeed, 46% of patients in both the derivation and validation ATRIA cohorts were categorized by the ATRIA score as having <1% per year risk. Such low risk indicates a small net clinical benefit from anticoagulation., Importantly, all aspects of improved performance with the ATRIA score were reproduced in the separate, larger, and contemporary ATRIA‐CVRN cohort of incident AF patients.

26When we restricted outcome to severe events, model discrimination improved, resulting in a c‐index of 0.76 for the ATRIA cohort. Clinically, prevention of severe events is more important than prevention of minor events and accurate prediction of severe events more relevant to the anticoagulation decision. Our findings should prompt other investigators developing stroke risk models for AF to also focus on severe outcomes.

27All AF patients who have had an ischemic stroke are at high risk of future stroke, and anticoagulation is strongly indicated. Risk prediction in primary prevention is more uncertain and where most of the challenge of anticoagulation decision‐making lies. Our ATRIA model provides a separate scoring scheme for primary prevention, and this scheme also performed better than the comparator schemes, a finding that was replicated in the separate, external ATRIA‐CVRN cohort. Performance of the ATRIA score was particularly good in predicting severe events in the primary prevention cohort.

28Recent reports highlight the promising performance of biomarkers in predicting stroke in patients with AF. Our ATRIA score appears to provide an improved clinical risk factor model on which to add such biomarkers with the goal of optimal risk prediction.

29Our model development had notable strengths. We based our findings on a generalizable community‐based cohort with a wide range in age and comorbidities representative of patients treated in usual care settings. The study sample was large, with hundreds of physician‐validated outcome events, allowing powerful assessment of the limited number of candidate predictor variables we tested. Case record review allowed us to assign Rankin severity ratings. We employed a split‐sample derivation/validation approach, and we used multiple bootstrap sample model‐building to reduce overconfidence in the selection of predictors. Most importantly, we validated our model and its relative performance versus the 2 leading alternative risk scores in an independent, contemporary, “external” validation cohort—the ATRIA‐CVRN inception AF cohort. Testing in additional data sets will assess the broader generalizability of our findings. Although the ATRIA score is more complicated than commonly used schemes, particularly CHADS2, the increasing availability of automated real‐time decision support tools based on data routinely available in electronic medical records will likely reduce the need for a simple scoring system.

30The ATRIA stroke risk score for AF patients was rigorously developed in a community‐based cohort and externally validated in a separate recently assembled community‐based AF cohort. Our results highlight the importance of including multiple categories of age, prior stroke, and their interaction in determining stroke risk in AF patients. The ATRIA score's performance was superior to the widely used CHADS2 and CHA2DS2‐VASc risk schemes in terms of c‐index and NRI. It identified a substantially larger fraction of patients at low stroke risk. Its performance was particularly good in predicting severe strokes, the category of stroke we are most interested in avoiding. Finally, the ATRIA score lends itself to calculating risk in primary prevention patients, the large group whose stroke risk is most uncertain and for whom personalizing the anticoagulation decision is most pressing. The ATRIA score provides an improved clinically based model on which to add informative biomarkers.