Computerized clinical decision support systems for chronic disease management: A decision-maker-researcher partnership systematic review

1Chronic conditions present patients, practitioners, and healthcare systems with some unique demands, including recurrent visits, adherence to complex care plans, long-term disease and treatment monitoring, behavior modification, and patient self-management. For the many patients with multiple co-morbidities, overlapping or diverging care plans may further complicate these processes.

2Computerized clinical decision support systems (CCDSSs) may help practitioners meet the requirements of chronic care. These systems analyze a patient's characteristics to provide tailored recommendations for diagnosis, treatment, patient education, adequate follow-up, and timely monitoring of disease indicators. For example, Holbrook et al. gave providers and diabetic patients access to a web-based system that offered care advice, allowed monitoring of diabetes risk factors, and tracked key care targets. As with any health intervention, however, rigorous testing is warranted to determine whether CCDSSs improve chronic care processes and patient outcomes.

3In our previous review of the effects of CCDSSs, we analyzed 100 randomized and non-randomized studies published until September 2004, 40 of which assessed the effects of CCDSSs on disease management. Of these 40 studies, 37 measured processes of care of which 62% (23) showed an improvement, and 27 measured patient outcomes of which 19% (5) showed an improvement. The quality of the studies varied widely, but improved over time.

4Many new randomized controlled trials (RCTs) have been published in this field since our previous work, potentially documenting important advances. Recognizing that the management of chronic disease has unique characteristics, we wished to review the impact of CCDSSs on the quality and effectiveness of chronic care. We had the opportunity to include the perspectives of senior hospital managers and front-line healthcare practitioners to ensure that relevant data were extracted and summarized--a level of stakeholder engagement that has not been included in other reviews.

5We previously published the details of our review protocol, openly accessible at http://www.implementationscience.com/content/5/1/12. These methods are briefly summarized here, along with details specific to this review of CCDSSs for chronic disease management.

6Do CCDSSs improve chronic disease management processes or patient outcomes?

7We conducted this review in partnership with individuals responsible for implementing CCDSSs in our region. Decision makers, both managers and clinicians, met with the review team periodically to discuss direction and specific details for the data extraction, analysis, presentation and interpretation of results.

8Full details of our search strategy are in our review protocol. In summary, we searched MEDLINE, EMBASE, Ovid's Evidence-Based Medicine Reviews, and Inspec until 6 January 2010, and reviewed the reference lists of included RCTs and relevant systematic reviews. We screened articles for eligibility in two stages: a duplicate, independent review of titles and abstracts followed by a duplicate, independent, full-text review of potentially eligible articles, with a third reviewer resolving disagreements.

9We selected RCTs of a CCDSS used by a health care provider for management of chronic conditions, published up to 6 January 2010 in any language that measured CCDSS impact on processes of care or patient outcomes. We included RCTs in any language that compared patient care with a CCDSS to routine care without a CCDSS and evaluated clinical performance (i.e., a measure of process of care) or a patient outcome. Additionally, to be included in the review, the CCDSS had to provide patient-specific advice that was reviewed by a healthcare practitioner before any clinical action. Studies were excluded if the system was used solely by students, only provided summaries of patient information, provided feedback on groups of patients without individual assessment, only provided computer-aided instruction, or was used for image analysis. Trials included in our previous review were included if they were eligible. Trials of CCDSSs for managing narrow therapeutic index medications used in some chronic conditions (such as warfarin in atrial fibrillation) were not included in this review, but are discussed in our review for therapeutic drug monitoring and dosing.

10To meet the needs of our management and clinical partners, we extracted study characteristics (e.g., study design, size, setting, authorship, funding, and year of publication) and system characteristics (e.g., integration with other systems, user interface elements, methods of data entry and delivery of recommendations, target users, and implementation details such as pilot testing and user training). Disagreements were resolved by a third reviewer or by consensus. We contacted primary authors to provide missing data and to assess the accuracy of the extracted data; 78% (43/55) provided input. For the remaining trials, a trained reviewer assessed the extraction form against the full-text to confirm accuracy.

11Using a 10-point scale, pairs of reviewers independently evaluated the selected trials on five dimensions of quality, including concealment of allocation, appropriate unit of allocation, appropriate adjustment for baseline differences, appropriate blinding of assessment, and adequate follow-up. We used a 2-tailed Mann-Whitney U test to compare methodologic scores between trials published before the year 2000 and those published later to determine if trial quality has improved with time.

12We assessed the effectiveness of CCDSSs in each trial for improving process of care and patient outcomes. We defined process outcomes as changes in care activities such as diagnosis, treatment, and monitoring of disease. Examples of patient outcomes included changes in blood pressure, clinical events and health-related quality of life. We judged a CCDSS effective if it produced a statistically significant (p < 0.05) improvement in a primary chronic disease outcome or in ≥50% of multiple relevant pre-specified outcomes. We considered primary any outcome that trial reports described as 'primary' or 'main.' If authors did not designate a primary outcome, we considered the outcome used to calculate the trial's sample size to be primary, if reported. When there were no pre-specified outcomes, the system was considered effective if it produced an improvement in ≥50% of all reported chronic disease outcomes. Our assessment criteria are more specific than those used in our 2005 review; therefore, the assignment of effect was adjusted for some trials included in the review.

13We summarized data using proportions, medians, and ranges. Denominators vary in some proportions because not all trials reported relevant information. All analyses were carried out using SPSS, version 15.0. We did not attempt a meta-analysis because of study-level differences in participants, clinical settings, disease conditions, interventions, and outcomes measured.

14We conducted a sensitivity analysis to assess the possibility of biased results in studies with a mismatch between the unit of allocation (e.g., clinicians) and the unit of analysis (e.g., individual patients without adjustment for clustering). We compared success rates between studies with matched and mismatched analyses using chi-square for comparisons. No differences in reported success were found for either process of care outcomes (Pearson X2 = 1.41, p = 0.24) or patient outcomes (Pearson X2 = 1.45, p = 0.23). Accordingly, results have been reported without distinction for mismatch.

15Figure 1 shows a flow diagram of included and excluded trials. We identified 166 trials of CCDSSs and Cohen's κ for reviewer agreement on trial eligibility was 0.93 (95% confidence interval [CI], 0.91 to 0.94). In this review, we included 71 publications describing 55 trials (33% of total) about management of chronic diseases. Thirty-eight included studies contributed outcomes to both this review and other CCDSS interventions in the series; three studies to four reviews, 12 studies to three reviews, and 23 studies to two reviews; but we focused here on outcomes relevant to the management of chronic disease.

16Summary of trial quality is reported in Additional file 1, Table S1; system characteristics in Additional file 2, Table S2; study characteristics in Additional file 3, Table S3; outcome data in Table 1 and Additional file 4, Table S4; and other CCDSS-related outcomes in Additional file 5, Table S5.

17Additional file 1, Table S1 presents details of our methodological quality assessment. Of the 55 trials, 53% reported adequate concealment of allocation; 78% showed no differences in baseline characteristics between study groups or adjusted accordingly; 53% allocated entire wards or practices to each study group; all except one used objective outcomes or blinding of outcome assessments; and 60% achieved a ≥90% follow-up rate for their unit of analysis. The overall quality of trials was good (median methods score, 8; ranging from 2 to 10) and improved with time (median methods score before versus after year 2000, 7 versus 8, 2-tailed Mann-Whitney U = 137; p = 0.005), possibly because early trials often failed to conceal allocation or to achieve adequate follow-up.

18Additional file 2, Table S2 describes CCDSS design and implementation characteristics. Denominators vary because not all trials reported on all features considered. Fifty-nine percent (32/54) of CCDSSs were integrated with electronic medical records, and 17% (8/47) were also integrated with computerized physician order entry systems. Fifty-three percent (25/47) automatically obtained data needed to give recommendations from electronic medical records; 36% (17/47) relied on practitioners to enter the data; and 26% (12/47) used research staff for this purpose. Advice was provided at the time of care in 85% of trials (46/54) most often on a desktop or laptop computer (51%; 26/51) or by existing non-prescribing staff (22%; 11/51). Fifty-three percent (29/54) provided advice to other healthcare practitioners in addition to physicians and 15% (8/55) directly advised patients in addition to practitioners. Sixty-four percent (25/39) of systems were pilot tested and healthcare professionals were trained to use them in 72% (34/47). Reports rarely described the CCDSS user interface characteristics.

19Seventy-three percent of trials (40/55) declared that at least one author was involved in the development of the system and three trials indicated that all authors were independent of development.

20Additional file 3, Table S3 provides further details of the CCDSS intervention, care setting, study funding source, and year of publication. Trials included a total of 7,335 practitioners (median, 72; ranging from 5 to 1,378 [when reported]) caring for 381,562 patients (median, 719; ranging from 27 to 156,772 [when reported]) in 974 clinics (median, 13; ranging from 1 to 112 [when reported]) across 705 distinct sites (median, 4; ranging from 1 to 112 [when reported]). Eight trials did not report their source of funding. Of the remaining 47, 74% (n = 35) were publicly funded, 17% (n = 8) were conducted with only private funds,, and 9% (n = 4) were conducted with a combination of private and public funding. The earliest trial was published in 1977, but over one-half (62%) were published after our previous search in September 2004.

21Table 1 summarizes the effects of all systems for improving process of care and patient outcomes and Additional file 4, Table S4 provides further detail regarding systems and individual outcomes selected for evaluation.

22Eighty-seven percent (48/55) of trials measured effects on chronic disease management processes, and 52% (25/48) demonstrated improvement. Sixty-five percent (36/55) measured impact on patient outcomes and 31% (11/36) of these demonstrated benefit on measures such as health-related quality of life, rates of hospitalization, unscheduled care visits, and a host of disease-specific clinical outcomes.

23Thirteen trials described systems primarily supporting diabetes care (median quality score, 7; ranging from 2 to 10). Fifty-five percent (6/11) reported improvements in processes of care including treatment and monitoring, while 62.5% (5/8) reported improvements in corresponding patient outcomes including blood pressure, HbA1c, and low-density lipoprotein (LDL) cholesterol. The seven trials published since 2005 appeared to show success more consistently: four of five improved the process of care, and five of seven improved patient outcomes.

24Systems in five diabetes trials targeted patients in addition to practitioners. Of these, all four trials that measured process effects demonstrated benefit, and four reported improvement in patient outcomes.

25Several recent trials were conducted in primary community clinics whereas most previous trials were conducted in hospitals. For example, in two trials conducted across multiple practices, CCDSSs provided patient-specific reminders during visits and notified at-risk patients of their care targets and upcoming appointments. Both trials demonstrated improvements in composite process measures comprising timely completion of foot and eye exams, and monitoring of blood pressure, HbA1c, lipoproteins, and renal function. Both trials also showed improvements in corresponding composite patient outcomes.

26CCDSSs in five trials (median score, 7; ranging from 6 to 8) provided recommendations for a host of conditions in conjunction with diabetes, including dyslipidemia, hypertension, obesity, and heart failure. Their effects on diabetes outcomes could not be isolated. All five measured process of care, and 80% (4/5) found improvements. Only one measured corresponding patient outcomes, but showed no benefit.

27The 10 trials focusing primarily on hypertension management (median score, 7; ranging from 4 to 10) were older, with 70% (7/10) published before 2005.

28Eight of 10 trials assessed impact on process of care using measures such as adherence to recommendations for blood pressure control, patient satisfaction, and number of scheduled care visits, and four demonstrated improvements.

29In contrast to diabetes systems, however, hypertension systems showed little or no patient benefit. Of the nine trials that reported patient outcomes, such as blood pressure and health-related quality of life, only one found benefit. This multi-component system improved patients' perceived health status by giving suggestions for the management of hypertension, obesity, and renal disease. The trial, however, was of poor quality (methods score 4), and the nature of the intervention prevented isolating effects related to hypertension.

30Four trials evaluated systems that focused primarily on dyslipidemia. All were conducted in primary care settings and published after 2005 (median quality score, 8.5; ranging from 7 to 10).

31Three trials measured effects on process of care and demonstrated improvements in lipid monitoring and treatment, but only one of three trials measuring patient outcomes found a benefit. This CCDSS generated patient-specific reminders that were mailed to primary care physicians and nurses; highlighted the patient's risk factors, lipoprotein values, and current medications; and recommended initiation or adjustment of lipid-lowering treatment when appropriate. The trial detected improvements in blood lipid monitoring and treatment management, as well as relative reductions in patients' LDL cholesterol.

32The nine trials of systems supporting asthma care were of excellent quality (median score, 9; ranging from 8 to 10) and relatively new (7/9 published after September 2004), but the systems were generally ineffective. All trials measured effects on process of care (including rates of spirometry, thorax radiography, IgE levels, and allergy testing; medication prescriptions and influenza vaccinations; and use of rescue medications) but only one demonstrated benefit.

33Two of five trials measuring patient outcomes found an impact. One system delivered asthma recommendations in primary care, made prognostic predictions by matching patients to similar known cases, and allowed users to print self-management plans for their patients. The trial demonstrated a reduction in acute asthma exacerbations and patient-initiated primary care visits. Another system delivered guideline recommendations to general practitioners and pneumologists, and proved to be more cost effective at improving quality of life than usual asthma care.

34Three asthma systems also gave advice for management of COPD. All of these measured process of care but detected no effects. One trial also measured patient outcomes but did not show benefit.

35Systems in four methodologically strong trials (median score, 9.5; ranging from 8 to 10) focused on heart failure, cardiac rehabilitation, ischemic heart disease, and angina. All measured process of care using adherence to guideline recommendations but only one found benefit. The CCDSS for cardiac rehabilitation used electronic medical records and needs assessment data to generate recommendations for exercise training, education, lifestyle change, and stress management. The trial demonstrated improved guideline adherence, but patient outcomes were not studied. The other three trials measured effects on quality of life as a patient outcome, but none found benefit.

36We did not group the remaining 12 trials due to their diverse primary indications. They focused on urinary incontinence, cancer, osteoporosis, renal disease, functional deficits, obesity, dementia, rheumatoid arthritis, advance directives, and various non-specific indications. Most trials found improvements in care process but only two demonstrated benefit to patients: one reduced urinary incontinence in nursing home patients, and the other improved likelihood of remission in patients with early rheumatoid arthritis through CCDSS-guided management of methotrexate.

37Four trials used cost-effectiveness as an outcome (see Additional file 4, Table S4). Only one trial demonstrated improvement to patient outcomes overall, and the CCDSS was also more cost-effective than usual asthma care.

38Additional file 5, Table S5 summarizes cost-related findings of the 12 trials that statistically compared costs of care between the study groups: six reported no difference with CCDSS compared to usual care, four reported savings with the CCDSS, and two reported that the CCDSS increased some costs.

39In addition to process of care and patient outcomes, we looked for effects on user satisfaction and workflow (see Additional file 5, Table S5). Only seven trials reported a formal effort of assessing user satisfaction: 3 found users satisfied overall, one found them unsatisfied, and the remaining three showed mixed results. The authors of five other studies commented that users were satisfied in informal evaluations.

40Two trials made formal attempts to measure systems' impact on user workflow and reported mixed results.

41This review was done in partnership with key decision makers to summarize the effectiveness of clinical decision support technology for the management of chronic conditions. We considered studies 'positive' if they showed a statistically significant improvement in at least 50% of relevant outcomes. CCDSSs often improved the process of patient care. When assessed, effects on any patient outcomes were rarely found, but may have been underestimated: 56% of trials reporting these outcomes declared them primary, and the remaining trials may not have been large enough or long enough to detect such outcomes. No study showed convincing evidence of benefit for major patient outcomes.

42Nevertheless, results from recent diabetes management trials are encouraging. Several of these systems were deployed in general community practice and those that engaged both patients and providers were consistently effective. These systems may become increasingly popular with the advent of patient-controlled electronic medical records. Systems addressing several conditions, including but not limited to diabetes, generally improved care but only one measured patient outcomes (no effect). In dyslipidemia, systems improved lipid monitoring and treatment, but only one reduced blood lipids. The few dyslipidemia trials were recent and may represent a promising area for future research.

43Conversely, most trials in hypertension measured patient outcomes and almost never found benefits, and only some showed improvements in the process of care. Asthma and COPD systems mostly failed to show effectiveness, despite being tested in recent, high-quality trials. The small collection of trials in heart failure, ischemic heart disease, cardiac rehabilitation, and angina also rarely show effects, with improvement only in rehabilitation processes. The remaining systems, too diverse to group, often improved care processes but were seldom found to benefit patients.

44While systems in diabetes appear to achieve success with respect to patient outcomes more often than systems in asthma and hypertension, we did not pre-specify this comparison and, given the play of chance and many possible confounders, we cannot confidently assert that the pattern is real. It is plausible that the effectiveness of CCDSS recommendations at improving patient outcomes for some indications is limited by the absence of high-quality clinical evidence in that area. Even the most scientifically sound recommendations, however, will fail to improve health outcomes if patients do not adhere to prescribed treatments--a very common problem. Unfortunately, our suggestions regarding the discrepancy remain purely speculative because studies did not explore reasons for failure, and we do not have enough trials to test these hypotheses reliably.

45The growing use of CCDSSs and their potential for benefit and harm highlight the importance of evaluating these systems in well-conducted randomized clinical trials. The increase in number and quality of trials is encouraging, but results remain mixed, and few trials investigated the mechanisms behind their findings. Careful description of study and system design in trial reports, as well as assessments of effectiveness and acceptability of system features, would support progress in this area.

46CCDSSs may represent a cost-effective way of improving chronic disease outcomes. However, the economic effects of systems are not readily assessed based on available data. The costs of design, local implementation, ongoing maintenance, and user support can be high, and may be further elevated by the unique nature of chronic care. This warrants cost-effectiveness analyses, but only four trials reported such data and little cost data of any kind are available across studies. If cost savings exist, however, current results suggest that they are modest.

47The benefits we can expect from the use of computerized decision support are not clear. Policy makers promoting the use of CCDSS, as well as healthcare administrators and practitioners considering local implementation, should be aware that the evidence of CCDSS effectiveness is limited, especially with respect to the small number and size of studies of patient outcomes. Further, evidence of benefit comes mainly from a few 'trail blazer' institutions with much in-house informatics expertise, evaluating home-grown systems developed over many years. As a result, trials in this review may not represent the effects in less technically endowed settings or from commercially available systems, the capabilities of which have been shown to vary greatly.

48Our review has some potential limitations. Great heterogeneity in CCDSS design, purpose, and targets for evaluation prevented us from conducting a meta-analysis. Instead, we used a binary measure of effect, where we considered studies 'positive' if they showed a statistically significant improvement in at least 50% of relevant outcomes. Thus, some of the studies we considered to show no effect found improvement on a minority of secondary or non-prespecified outcomes. These findings could be real but could also be due to post hoc unplanned analyses and multiple testing. Readers should refer to the Methods section for a more detailed account of our effect assessment.

49We were unable to assess the risk of publication bias in this literature. Given that most systems were studied by their own developers, we suspect that publication bias is likely, and even our findings of modest effects may overestimate the true likelihood of seeing benefit from CCDSSs.

50Our method of summarizing the evidence by vote counting inflates the risk of Type 2 error and should generally be approached with caution. However, our results remain essentially unchanged from our 2005 review and are comparable to another major review conducted by Kawamoto and colleagues, and a recent 'umbrella ' review of high-quality systematic reviews of CCDSSs in hospital settings. Another recent review of reminder systems (a subset of CCDSS) summarized evidence by effect size meta-analysis and qualified the impact of these interventions as falling below the thresholds of clinical importance. Given the similar conclusions of these other systematic reviews and the risk of publication bias in the CCDSS literature, we have little reason to believe that our methods underestimate the benefit from these systems.

51Finally, we observed improvements in the quality of trials over time but this trend may have resulted from better reporting in more recent studies.

52CCDSSs can improve chronic disease management processes and, in some cases, patient outcomes. Recent trials in diabetes care show the most promising results. The mechanisms behind systems' success or failure remain understudied. Future trials with clear descriptions of system design, local context, implementation strategy, costs, adverse outcomes, user satisfaction, and impact on user workflow will better inform CCDSS development and decisions about local implementation.

53RBH, PSR, SM, HCG, AXG, RJS, JAM, NMS, LWK, and TN received support through the Canadian Institutes of Health Research Synthesis Grant: Knowledge Translation KRS 91791 for the submitted work. PSR was also supported by an Ontario Graduate Scholarship, a Canadian Institutes of Health Research Strategic Training Fellowship, and a Canadian Institutes of Health Research 'Banting and Best' Master's Scholarship. Additionally, PSR is a co-applicant for a patent concerning computerized decision support for anticoagulation, which was not discussed in this review, and has recently received awards from organizations that may benefit from the notion that information technology improves health care, including COACH (Canadian Organization for Advancement of Computers in Healthcare), the National Institutes of Health Informatics, and Agfa HealthCare Corp. RJS is the owner of Fig.P Software Incorporated, which develops and sells a chronic disease management system that is not a subject of this review. HCG has/had financial relationships with the following organisations in the previous three years: Sanofi Aventis, GlaxoSmithKline, Eli Lilly, Novo Nordisk, Astra Zeneca, BMS, Roche, Bayer, Janssen Ortho, Solvay, BI, Servier. RBH is acquainted with several CCDSS developers and researchers, including authors of papers included in this review.

54RBH was responsible for study conception and design; acquisition, analysis, and interpretation of data; drafting and critical revision of the manuscript; obtaining funding; study supervision. He is the guarantor. PSR acquired, analyzed, and interpreted data; drafted and critically revised the manuscript; and conducted statistical analysis. SM acquired data; drafted and critically revised the manuscript. HCG analyzed and interpreted data; and critically revised the manuscript. AXG acquired, analyzed, and interpreted data; and critically revised the manuscript. RJS analyzed and interpreted the data. JAM acquired, analyzed, and interpreted data; and critically revised the manuscript. LWK and TN acquired data and drafted the manuscript. NLW acquired, analyzed, and interpreted data; provided administrative, technical, or material support; and provided study supervision. All authors read and approved the final manuscript.