Do cognitive interventions improve general cognition in dementia? A meta-analysis and meta-regression

1This is a comprehensive meta-analysis of cognitive interventions in Alzheimer's disease (AD), specifically examining efficacy of interventions compared to active and non-active control groups.

2By examining common clinically used general cognitive outcome measures, we question whether cognitive interventions lead to clinically important differences.

3This meta-analysis highlights important limitations in the literature such as difficulties with blinding of patients and use of adequate placebo controls, which make comparison with the results of dementia drug trials problematic.

4Cognitive interventions are widely used to aid cognitive function in people suffering from dementia. There are three main approaches, which have been summarised by Clare and Woods. Cognitive training (CT) involves repeated practice of a standardised task that targets a specific cognitive function. The assumption is that such ‘training’ will lead to an improvement in the cognitive domain trained, and potentially to generalised improvements in cognitive function. Such CT is usually delivered individually, and may be computerised or non-computerised. CT is often adaptive, allowing an increase in task difficulty as expertise develops.

5Cognitive stimulation (CS) refers to a more non-specific approach, where a range of different activities are used to engage and stimulate the individual. There may be components of reminiscence therapy, reality orientation, social activity and sensorimotor activities. Emphasis is on the involvement of multiple cognitive domains rather than the targeting of one specific cognitive function. It is normally a group rather than individual intervention, with a significant emphasis on social interaction.

6Cognitive rehabilitation (CR) differs in that it takes a particular impaired ability as the starting point and, using a person-centred approach, seeks to find solutions or approaches that enable the individual to perform the desired function or task (table 1).

7The National Institute for Health and Care Excellence (NICE) guidelines recommend the use of CS, however, there is a lack of clarity over the effectiveness of these interventions in terms of stabilisation or improvement in cognition. A Cochrane meta-analysis of CS included 15 randomised controlled trials (RCTs) and concluded that CS significantly improved general cognitive outcomes such as the mini-mental state examination (MMSE, mean difference=1.74, 95% CI 1.13 to 2.36, p<0.001) and Alzheimer's disease Assessment Scale-Cognition (ADAS-Cog, mean difference=2.27, 95% CI 0.99 to 3.55, p=0.0005). A Cochrane review of 12 RCTs investigating CT or CR reported no significant improvements on any cognitive outcome measure. Neither of these meta-analyses examined the effects of including active or non-active control conditions on effect size.

8By contrast, Sitzer et al reviewed 5 non-RCTs and 12 RCTs of cognitive interventions in dementia, defined by whether compensatory or restorative approaches were used. Overall effect sizes of (Cohen's) d=0.37 (SD 0.45) for restorative and d=0.40 (SD 0.46) for compensatory interventions on general cognitive outcomes were reported. Studies that compared intervention to waiting list controls tended to produce greater effect sizes (d=0.53, SD=0.47) than those using attention-controlled placebo controls (d=0.36, SD=0.58), although this difference in effect size was non-significant (p=0.511).

9Most recently, Kurz et al found significant standardised mean differences (SMD) on the MMSE (SMD 0.21, 95% CI 0.03 to 0.39, p=0.02) and ADAS-Cog (SMD −0.3, CI −0.48 to −0.13, p=0.0005) for CS, but not for CT and CR in a meta-analysis of RCTs in dementia and mild cognitive impairment (MCI). These authors concluded that there was no convincing evidence that these cognitive score changes generalised to any clinically significant improvements in quality of life or activities of daily living (ADLs).

10Consideration of these meta-analyses highlights the limitations of the evidence base. Methodological difficulties, such as a lack of suitable control interventions and failure to maintain complete blinding to allocation, can mean that factors such as increased attention, socialisation or motivation could contribute to observed changes, or that participant and investigator placebo effects may operate. This would be particularly expected for CS interventions which are often less specific in nature and encompass more social activities.

11The current analysis therefore aimed to: evaluate the efficacy of cognitive interventions with consideration of the use of ‘active’ and ‘non-active’ controls. Active controls comprise of interventions designed to control for non-specific therapeutic effects, including time, attention and non-specific input from research or clinical teams (eg, social support, psychoeducation, discussion groups or non-directed activities). Non-active controls consist of treatment as usual (TAU), waiting list conditions, or a minimal intervention not matched for time, social interaction or with no specific cognitive content (table 1);examine effects on commonly-used clinical outcomes of general cognitive function (MMSE and ADAS-Cog) and consider whether these met published criteria for minimum clinically important differences (MCIDs);conduct meta-regression analyses to examine associations between effect sizes and variables that may influence the efficacy of cognitive interventions, such as format, setting or intensity of intervention, severity of dementia or study quality.

12evaluate the efficacy of cognitive interventions with consideration of the use of ‘active’ and ‘non-active’ controls. Active controls comprise of interventions designed to control for non-specific therapeutic effects, including time, attention and non-specific input from research or clinical teams (eg, social support, psychoeducation, discussion groups or non-directed activities). Non-active controls consist of treatment as usual (TAU), waiting list conditions, or a minimal intervention not matched for time, social interaction or with no specific cognitive content (table 1);

13examine effects on commonly-used clinical outcomes of general cognitive function (MMSE and ADAS-Cog) and consider whether these met published criteria for minimum clinically important differences (MCIDs);

14conduct meta-regression analyses to examine associations between effect sizes and variables that may influence the efficacy of cognitive interventions, such as format, setting or intensity of intervention, severity of dementia or study quality.

15Online literature databases and trial registers (Web of Knowledge, Cochrane Collaborative Central Register of Controlled Trials, and PubMed/Medline) were searched on 6 June 2013 using the terms in table 2. Previous meta-analyses and systematic reviews of cognitive interventions in dementia were also searched, in addition to leading journals.

16Studies were included in the meta-analysis if the study was a peer-reviewed RCT; participants had a diagnosis of dementia; mean age of participants in the study was greater than 60 years; sufficient data were available for calculation of effect sizes (unavailable information was requested from authors and included if obtained); the number of participants in each condition was more than 5 at any point, and standardised general cognitive outcome measures were used. RCTs were included if they compared a cognitive intervention to an active or non-active control or with another treatment (pharmacotherapy or other non-pharmacological therapy). Cognitive interventions were classified as CT, CS or CR as described in the introduction and in table 1. Studies were independently screened and selected for inclusion and rated as to the best description of the intervention and control groups by three of the authors (JDH, KL, MS), using the criteria described in table 1. If it was decided that a study contained elements of more than one type of intervention it was classed as mixed for example, mixed CT and stimulation (MCTS). Control interventions were classed as either ‘active’ or ‘non-active’ as described in table 1. Disagreements were resolved through discussion with the fourth and fifth authors (RLG and RJH).

17A risk of bias tool was used to assess study quality in five areas known to affect clinical outcomes (sequence generation, allocation concealment, blinding of outcome assessors, incomplete outcome data and selective outcome reporting). Studies were independently and blindly rated as to the degree of bias in each study by three of the authors (JDH, KL and MS) and disagreements resolved through discussion with a fourth author (RLG). If a study received an inadequate or unclear rating in all five areas of bias it was excluded from meta-analyses.

18Means and SDs or SEs for each general cognitive outcome measure in each condition and time point were independently extracted for each study by three of the authors (JDH, KL, MS). Disagreements were resolved through discussion with a fourth author (RLG). If means and SD were not available in published articles, authors were contacted and obtained information was included.

19Effect sizes (Hedges’ g) were calculated by computing the mean change scores (Mpost−Mpre or Mfollow-up−Mpre) between the intervention and comparator conditions (control or other treatment groups), which allows an estimate of effectiveness even when the intervention and control groups are non-equivalent. The mean change scores were divided by the pooled estimates of the intervention and comparator SDs at preintervention (SDpre) and corrected for positive bias (Cp) to account for bias resulting from small sample sizes. Further details of effect size calculations are found in online supplementary appendix 1.

20Separate meta-analyses were conducted for subtype of cognitive intervention (CT, CS, CR and MCTS), in combination with subtype of control group (active or non-active) and general cognitive outcome measure (MMSE, ADAS-Cog or other), to provide specific pooled effect sizes for each type of intervention and outcome. Separate meta-analyses were also conducted for each outcome measure at different time points (postintervention (defined as 0–4 weeks after the intervention), 3, 6 and 9–12-month follow-up) to avoid non-independence of effect sizes.

21Planned meta-regression analyses were used to examine whether any between-study heterogeneity could be explained by format of intervention (group or individual) and measures of study quality (sequence generation, allocation concealment, blinding of outcome assessors), as these have been suggested by previous analyses to influence effect size. Other variables examined were setting of intervention (outpatient/community vs inpatient/care home facilities), intensity of intervention (hours per week), length of intervention (weeks) and severity of dementia (as determined by mean MMSE score).

22Mean change scores for each study were also calculated as (Mpostintervention−Mpreintervention)−(Mpostcomparator −Mprecomparator) to provide estimates that could be directly compared with MCID in outcome measures. The MCID for interventions in dementia has been systematically reviewed elsewhere. For the ADAS-Cog, the most commonly cited measure, there is general agreement that a 4 point change is clinically significant. There is a greater range of opinion for the MMSE, with values of between 1.4 and 3 being cited. Further details of all statistical analyses are found in online supplementary appendix 1.

23Literature searches identified 2206 potential studies, 59 of which met inclusion criteria for data extraction (PRISMA flow diagram, figure 1 and PRISMA checklist, online supplementary table S1). Of these, 33 contained general cognition outcome measures that could be included in meta-analyses and were selected. Summary characteristics of these studies are presented in online supplementary table S2.

24Four studies were classified as CT, 21 as CS, and 7 as MCTS. One study contained separate CS and MCTS interventions. There were no RCTs of CR with general cognitive outcomes.

25Only eight studies used active control groups. Twenty-one studies used non-active control groups, while two studies had active and non-active control groups. One study compared the intervention to other treatments and non-active controls, and one study compared the same intervention in different settings.

26The most commonly used general cognitive outcome measure was the MMSE. Seventeen studies used the MMSE alone, 10 studies included the MMSE and ADAS-Cog and 2 studies used the ADAS-Cog alone as a general cognition outcome measure. Two studies used only other general cognitive measures (Clifton Assessment Schedule (CAS) and The Mattis Dementia Rating Scale (MATTIS)). Two studies used the MMSE and one other general cognitive measure Milan Overall Dementia Assessment (MODA).

27Only eight studies included follow-up data, ranging from 6 weeks to 10 months postintervention, with the most common follow-up period being 6 months.

28Risk of bias and study quality is summarised in online supplementary table S3. Randomisation was the least adequately addressed, with only 12 studies adequately or partially adequately reporting randomisation sequence and 10 studies adequately reporting allocation concealment.

29The results of all meta-analyses conducted are presented in table 3. Postintervention, there was a significant pooled effect size for CS versus non-active controls on the MMSE (g=0.51, 95% CI 0.35 to 0.66, z=6.23, p<0.001, figure 2). There was low heterogeneity between studies (I2=24.9%). The calculated 95% prediction interval (0.124 to 0.89) suggested that the intervention was beneficial in individual settings.

30A smaller but still significant pooled effect size of 0.35 (95% CI 0.06 to 0.64; z=2.34, p=0.019) was found for CS versus active controls on the MMSE (figure 3), with no heterogeneity between the three studies (I2=0.0%).

31On the ADAS-Cog there was a significant pooled effect size favouring CS of −0.26 (95% CI −0.44 to −0.08; z=2.82, p=0.005, figure 4). There was low heterogeneity between the nine studies (I2=18.5), however, 95% prediction intervals (−0.62 to 0.10) suggested that the intervention may not be beneficial in individual settings. There were no studies comparing CS to active controls that used the ADAS-Cog as an outcome measure.

32Two CS versus non-active control studies assessing other general cognitive outcome measures (CAS and MODA) were included in a meta-analysis. A non-significant positive effect size of 0.25 was found (95% CI −0.44 to 0.94; z=0.71, p=0.48).

33At up to 3 months follow-up, there was a pooled effect size of 0.796 in favour of CS (95% CI 0.052 to 1.539; z=2.10, p=0.036), on the MMSE, however, both studies included in the analysis compared CS to non-active controls. There was moderate heterogeneity between these studies (I2=54.5%). By 6 months follow-up a non-significant pooled effect size of 0.273 (95% CI −0.10 to 0.64; z=1.45, p=0.15) was found on the MMSE, with no heterogeneity between the three studies (I2=0.0%). At 10 months follow-up, a significant effect of CS (g=−0.40; 95% CI −0.723 to −0.075; z=2.41, p=0.016) on the ADAS-Cog was seen.

34Only one study compared CT to a non-active control group, therefore no meta-analyses could be conducted. On the MMSE there was a non-significant pooled effect size of 0.22 favouring CT versus active controls (95% CI −0.745 to 1.180; z=0.44, p=0.658). There was significant heterogeneity between the three studies (I2=6.9%) and 95% prediction intervals (−11.033 to 11.467) suggested that the intervention may not be beneficial in all individual study settings.

35There were no studies comparing CT to active or non-active control groups using the ADAS-Cog as an outcome measure. One study used the MODA as an outcome measure therefore no meta-analyses could be conducted.

36Non-significant pooled effect sizes were found with MCTS versus non-active, g=0.447 (95% CI −0.568 to 1.462; z=0.86, p=0.388) and active controls, g=0.253 (95% CI −0.179 to 0.686; z=1.15, p=0.251) on the MMSE. Heterogeneity between the three MCTS versus non-active control studies was significant (I2=73.8%) with 95% prediction intervals (−11.333 to 12.227) suggesting the intervention may not be beneficial in individual settings.

37It was not possible to conduct meta-analyses on studies comparing cognitive interventions to other treatments (eg, pharmacological treatment) as only a single study investigated this. Similarly only a single study compared a cognitive intervention in different settings and therefore no meta-analysis was performed.

38The results of the meta-regression analyses are presented in table 4.

39For MMSE and ADAS-Cog outcome measures, meta-regression analyses revealed no significant associations between effect sizes and type of control group (active vs non-active), setting (inpatient vs outpatient), length of intervention, format of intervention (group vs individual), intensity of intervention in hours per week, or mean severity of dementia of participants. In addition, there were no significant associations between effect sizes and measures of potential bias: randomisation sequence, randomisation allocation, blinding of outcome assessors and incompleteness of outcome data or selective outcome reporting.

40The limited number of studies precluded meta-regression analysis at any of the follow-up time points.

41Repeated random-effects meta-analyses of the main CS comparisons, using SDs of mean change scores, produced a significant weighted mean difference score of 1.78 (95% CI 1.23 to 2.33, p<0.001) for CS versus non-active controls on the MMSE and −1.92 (95% CI −3.43 to −0.4, p=0.01) for CS versus non-active controls on the ADAS-Cog. However, the weighted mean difference score for the MMSE in CS versus active controls was non-significant (1.45, 95% CI −0.11 to 3.02, p=0.07).

42Comparisons of the calculated mean change scores for each study with the range of published MCIDs are presented in table 5. For the CS studies, there was only evidence of the majority of studies (11/17) reaching minimal clinical significance with the lowest published threshold for MCID (1.4 MMSE points). However, with the more conservative MCID of >2 MMSE points, only 9/17 CS versus non-active studies and no CS versus active control studies reached MCID. Of note, only 2/9 CS versus non-active control studies reached MCID on the ADAS-Cog.

43There was evidence of statistically significant efficacy of CS when MMSE was used as the outcome measure, although effect sizes were small to moderate in magnitude (g=0.35 for active and 0.51 for non-active controls). ADAS-Cog score also showed significant improvement in comparisons with only non-active controls, again with a small effect size of g=−0.26. Where there was heterogeneity between the trials reviewed, prediction intervals indicated that CS was beneficial in individual settings as measured by the MMSE but not the ADAS-Cog, which questions the efficacy of CS on ADAS-Cog scores in individual study settings. Our meta-analysis is consistent with recent Cochrane reviews in finding little or no evidence for significant efficacy of CT in dementia, but we further conclude that interventions using a mixture of CT and CS approaches do not significantly improve general cognition.

44It is encouraging that for CS there was a statistically significant pooled effect size on general cognitive outcome measures. However, as the effect sizes from CS studies were only small to moderate in magnitude, can we be confident that statistically significant improvements on the outcome measures translate to clinically meaningful benefits in general cognition? Examination of between-group mean MMSE difference scores reveals that only when the lowest threshold for the MCID are used do the majority of studies (11/17 studies) reach minimal clinical improvement. The weighted mean difference for CS studies compared with adequate active controls of 1.45 (95% CI −0.11 to 3.02, p=0.07) only just reaches the lowest MCID threshold of 1.4 points, and was not statistically significant in the sensitivity analysis. When the ADAS-Cog was used as an outcome measure, only two of nine studies versus non-active controls (and no studies vs active controls) demonstrated mean differences of greater than four points, and the weighted mean difference for all studies of −1.92 (−3.43 to −0.4) lies well below the MCID of 4. As there was limited evidence of clinically important differences in MMSE or ADAS-Cog scores when CS was compared to an adequate placebo control, we would conclude that, although statistically significant improvements in MMSE or ADAS-Cog scores are seen with CS, our analysis is consistent with that of Kurz et al. We would similarly argue that there is only limited evidence that any cognitive interventions leads to clinically significant cognitive improvement in dementia. The definition of what constitutes a clinically significant difference, however, is open to debate. The MCID values quoted come from pharmacological intervention studies and are limited to the general cognitive outcome measures examined in this meta-analysis. Outcomes not examined in the current study, such as quality of life, functional improvement, mood and carer attitudes are all of definite clinical significance. However, it is valid to compare MCID on general cognitive measures when assessing interventions that aim to improve cognitive function, and therefore taken simply on general cognitive grounds alone, cognitive interventions remain of debatable clinical significance. Further exploration of the clinical significance of improving quality of life in dementia is needed, as well as further qualitative and quantitative analyses examining the effects of interventions to improve quality of life.

45A significant issue is the inadequacy of blinding and placebo controls in psychosocial RCTs such as those of CS. The gold standard for pharmacological interventions is the double-blind, active placebo-controlled RCT, with medications not adopted by NICE unless there is sufficient evidence from these types of RCTs. However, the same standards are not held for psychosocial interventions such as cognitive behavioural therapy (CBT) and CS. Consequently, psychosocial interventions may appear more effective than they truly are due to the overestimation of effect sizes resulting from inadequate blinding and placebo controls. Our results suggest this may be a factor, as larger effect sizes were found when CS was compared with non-active controls than when compared with active controls. The meta-regression found that type of control group was not significantly associated with ES; however, this may be due in part to the very small number of studies that used active control groups. Recent meta-analyses of psychological interventions in the elderly have demonstrated that significant effects are not present with active control groups compared with non-active controls.

46Appropriate control interventions need to be designed with care. If the intervention and control groups are too similar to each other, the study may be underpowered to detect an effect. Alternatively, if a control intervention is more enjoyable than the intervention, any true benefits of the intervention may be masked. However, the alternative of using only non-active or TAU control groups is more problematic, as it could lead to a difference being attributed to the intervention when it may have been due to the non-specific factors of contact, time, socialisation or motivation associated with the intervention.

47It is imperative that non-pharmacological interventions are subjected to the same rigour that pharmacotherapy interventions are before recommendations about efficacious treatments can be made, particularly in the current economic climate. At the very least studies examining the true efficacy of psychosocial interventions should aim to be single-blinded (ie, blinding of participants and outcome assessors, but not therapists), active placebo-controlled RCTs. It is, of course, difficult to blind participants in psychosocial interventions, and such blinding raises ethical issues, but it is not impossible, as demonstrated in a recent CBT trial. Although cognitive measures are performance-based, the knowledge that a participant is engaged in a placebo condition may have effects on engagement or motivation that could impact performance during the intervention and on outcome measures. There would be clear ethical issues in misleading participants into believing that a control is actually an intervention. However as the ‘active ingredients’ of efficacious cognitive interventions become clearer, a move to more comparative studies would be useful, with different training or stimulation programmes compared.

48Ultimately, it is clear from our meta-analyses that more randomised, single-blind, active placebo controlled studies are required to properly assess the efficacy of cognitive interventions in dementia.

49A difficulty in the literature lies in characterising the content of the interventions. We based our classifications on the definitions suggested by Clare and Woodsl, however, a wide range of differing content remained. Within CS, different approaches included reality orientation, reminiscence therapy, socialisation activities, use of external memory aids and physical exercise. As well as differing settings and formats, which could be quantified, there were also differences that were more difficult to quantify to allow formal analysis such as encouraging techniques to be practiced between sessions, motivation of participants, encouraging carer involvement and educating carers. We decided to include studies that appeared to use predominantly differing methods (eg, reality orientation and reminiscence therapy) within CS. It could be argued that it is not appropriate to directly compare studies with differing methodologies, or include studies that have a very wide range of interventions (eg, those that include physical and social as well as cognitive elements), due to the inability to isolate the active ingredient in these regimes. However, despite the wide variety of content, the studies included in our analyses all aimed to improve cognition by delivering a cognitive intervention and could be characterised according to the criteria stated in table 1. It could also be argued that some studies we have classed as mixed, could be classified as either CT or CS, as other authors have done. However, by creating a category of mixed CS and CT our aim was to reduce heterogeneity between studies, to improve the quality of the meta-analyses.

50A similar difficulty is in assessing the quality and nature of control interventions. There are a wide variety of control interventions in the literature and some may be criticised for being poorly structured, inconsistently delivered or not theoretically based. Placebo interventions that are poorly conceived, designed or delivered would clearly be inadequate controls. Therefore there is a need for theoretically based, well designed, blinded and adequate active control interventions, rather than relying on non-active TAU controls, which in themselves may differ widely from each other.

51In defining control groups as ‘active’ we acknowledge there may be heterogeneity between the active controls used in different studies. However, in our opinion, basing our classification on definitions in table 1, and analysing active controls separately from non-active controls provided a more precise estimate of the efficacy of cognitive interventions, than when all studies are included in the same meta-analysis irrespective of the type of control group.

52A further limitation of our study was only analysing MMSE and ADAS-Cog as outcome measures. This was based on the pragmatic need to examine clinically useful and recognised measures of general cognitive function. It is therefore beyond the scope of this meta-analysis to comment on whether a particular cognitive intervention may have significant benefits that could not be identified using the MMSE or ADAS-Cog. This is likely to apply to CT that may aim to improve function in a specific cognitive domain such as executive function, language or episodic memory. There may be benefits within these domains as a result of training that would be apparent if measured on more specific standardised tasks, but may not lead either to general cognition improvement, or improvements may not be seen on measures such as the MMSE or ADAS-Cog which may not be sufficiently sensitive to change. However, if these non-pharmacological interventions are to be recommended as ‘cognitive’ therapies for patients with dementia it is important to evaluate the evidence for their efficacy on overall cognition as well as on specific standardised tests.

53In conclusion, this meta-analysis and meta-regression has shown that CT or combined MCTS interventions do not improve general cognition in patients with dementia. There is evidence that CS programmes can improve MMSE and ADAS-Cog scores, however, heterogeneity means that CS may not show benefit on the ADAS-Cog in all settings, and improvements on the ADAS-Cog are not generally clinically significant. The limited number of studies that include adequate active control conditions and lack of double blinding also make it difficult to compare efficacy with pharmacological interventions.