Cognitive Training for Attention-Deficit/Hyperactivity Disorder: Meta-Analysis of Clinical and Neuropsychological Outcomes From Randomized Controlled Trials

1Attention-deficit/hyperactivity disorder (ADHD) is a childhood-onset condition characterized by pervasive patterns of inattention and/or impulsivity-hyperactivity that often persist into later life. Combinations of pharmacological and psychological approaches are recommended for its treatment. Although medication is efficacious in randomized controlled trials (RCT) in the short/medium-term and is indicated as the first-line treatment (at least for severe cases), it has a number of potential limitations—each affecting some patients. These include the following: partial response or nonresponse; possible adverse effects; uncertainty about long-term costs and benefits; poor adherence; and negative medication-related attitudes from patients, parents, or clinicians. Psychological treatments such as behavioral parent training are also widely used. However, a recent meta-analysis found no effects on ADHD symptoms when only ratings by assessors blind to treatment allocation were considered.

2In recent years, cognitive training has been investigated as a potential ADHD treatment. Building on evidence of brain plasticity from rehabilitation science and contemporary developmental neuroscience, cognitive training is premised on the notion that key brain networks implicated in ADHD can be strengthened, and the cognitive processes they subserve improved, through controlled exposures to information processing tasks. Thus, it is argued that cognitive training can reduce ADHD symptoms and improve functioning by targeting neuropsychological deficits thought to mediate ADHD pathophysiology. In keeping with the complex nature of ADHD neuropsychology, cognitive training approaches have targeted a range of deficits (e.g., attentional control, working memory, inhibitory control). Currently, such training is typically delivered via computers using adaptive procedures, whereby training task difficulty is automatically increased across sessions to continually challenge the patient at the boundaries of his or her competence. This has been shown in neuroimaging studies to be necessary for sustaining neuronal changes.

3The efficacy of cognitive training for ADHD was addressed in a meta-analysis of nonpharmacological treatments for ADHD by Sonuga-Barke et al. on behalf of the European ADHD Guidelines Group (EAGG). This meta-analysis focused solely on RCTs. Importantly, it addressed the issue of blinding by comparing outcomes rated by individuals most proximal to the therapeutic setting (often unblinded and invested in the patient and/or intervention) and those provided by reporters judged to be probably blinded. Effects of cognitive training on ADHD symptoms calculated using unblinded ratings were highly significant (standardized mean difference [SMD] = 0.64, 95% CI = 0.33–0.95). These effects dropped substantially (SMD = 0.24) and became statistically nonsignificant (95% CI = −0.24 to 0.72) when probably blinded measures were used. However, these results should be considered as preliminary because only 6 RCTs were included. The authors concluded that more evidence was required, especially from trials in which assessments were effectively blinded, before cognitive training could be supported as an ADHD treatment. A second meta-analysis by Rapport et al., published more recently and exploring a wider range of outcomes, found similar effects. However, compared to Sonuga-Barke et al., this more recent meta-analysis included only 2 additional peer-reviewed RCTs with outcomes related to ADHD core symptoms. Moreover, to increase statistical power, Rapport et al. also included non-RCTs and pooled across design types, making effect size estimates of the effects of cognitive training on ADHD core symptoms and related neuropsychological impairment difficult to interpret.

4A significant number of new RCTs of cognitive training for ADHD, not available for inclusion in these previous 2 meta-analyses, have been published in the past 2 years, reflecting the current interest in cognitive training in this field. The greater number of trials now available allows a much more definitive estimate of the effects of cognitive training to be made. In the present article, we update the first EAGG cognitive training meta-analysis to include these new trials, and we extend its focus to cover effects on neuropsychological processes and academic functioning, which were not addressed in the previous EAGG meta-analysis. The focus on neuropsychological processes is important for 2 reasons. First, neuropsychological deficits are postulated to mediate the pathways between originating causes and disorder onset: improvements in neuropsychological functioning may therefore be a prerequisite for ADHD symptom reduction. Second, they are associated with functional impairments in their own right, independent of their association with ADHD symptomatology, especially in social and academic contexts. A broad range of training approaches have been used with ADHD populations. In the meta-analysis by Sonuga-Barke et al., given the small number of studies available, trials with different techniques had to be pooled to generate an effect size estimate. However, given the increased number of trials now available, our aim was to explore training-type specific effects through the use of subanalyses where sufficient numbers of trials existed.

5The EAGG protocol for nonpharmacological interventions for ADHD was registered on the International Prospective Register of Systematic Reviews PROSPERO (http://www.crd.york.ac.uk/PROSPERO, protocol number: CRD42011001393). The same protocol was followed here.

6Only RCTs including interventions aimed to directly train a cognitive function were retained. As reported by the Cochrane group, to ensure high levels of methodological adequacy and to avoid the inevitable bias caused by dependence on investigators agreeing to provide data from unpublished studies, only published studies were included. Trials were included if participants had an ADHD diagnosis (any subtype) or met accepted cut-offs on validated ADHD rating scales and were between 3 and 18 years of age. Trials involving children with ADHD comorbid with rare disorders (e.g., fragile X syndrome) only were excluded. Control conditions allowed were “treatment as usual,” “wait list,” “active/placebo/sham” (i.e., involving other forms of computer-based activity or alternative training regimen). Trials were not excluded if patients received medication as part of normal treatment. In an extension of the EAGG protocol, trials could be included in this updated meta-analysis despite not reporting an ADHD outcome if they reported neuropsychological and/or academic outcomes.

7Sonuga-Barke et al. included studies up to April 3, 2012. Here, using the same search strategy, our final search date was May 18, 2014. Supplement 1, available online, reports details about the search strategy and syntax for each database. Parallel searches were conducted separately by the first 2 authors.

8For consistency with previous EAGG meta-analyses and to provide a robust estimate of effects, outcome domains were analyzed only if 5 or more RCTs were available. The outcomes analyzed were: ADHD symptoms (total ADHD as well as inattention and hyperactivity/impulsivity symptoms), parent ratings of executive functioning (e.g., Behavior Rating Inventory of Executive Function [BRIEF]), standardized measures of reading and arithmetic ability, and laboratory-based measures of verbal and visual working memory, inhibition, and attention. For neuropsychological outcomes, only scores from tasks different from those used for training were included in the analysis.

9Articles’ titles and abstracts were screened independently by the first 2 authors. Final inclusion was based on the full text. Trials were blindly double coded for eligibility by the first 2 authors (S.C., M.F). Disagreement was resolved by the senior author for 3 trials.

10Two authors independently assessed trial risk of bias using 5 domains of the Cochrane Collaborations tool: namely, selection bias, performance bias, detection bias, attrition bias, and other bias. If there was disagreement between the 2 raters, the final rating was established through consensus with the involvement of the senior author. This occurred for 4 trials.

11Trial information was entered into RevMan 5.0. Data extraction was independently performed and cross-checked by the first 2 authors. SMD was calculated as mean pre- to posttreatment change in the intervention group minus the mean pre- to posttreatment change in the control group divided by the pooled pretest standard deviation with a bias adjustment. SMDs for each trial were combined using the inverse variance method. Given the inherent heterogeneity of studies, random effects models were used. The I2 statistic was calculated a posteriori to estimate between-trial SMD heterogeneity. For the most proximal analysis, parent ratings, if available, were used for home-based interventions, and teacher ratings were used for school-based interventions, except when it could be inferred from the manuscript’s text that teachers were less blinded than parents for home-based interventions and parents less blinded than teachers for school-based interventions (2 trials). Probably blinded assessments were those made by an individual judged likely to be unaware of treatment allocation. In trials in which more than 1 such measure was available, the best-blinded measure was chosen. For home-delivered interventions, teachers’ ratings were usually judged to be blinded, whereas for school-based interventions, parents were judged to be blinded except where this could be inferred not to be the case from the text or from e-mail exchange with the authors. As per protocol, where direct observations were available, we selected these over rating-scale scores. This decision was based on the judgement that direct observations are likely, in general, to be better blinded than parent- or teacher-rated outcomes, even when the latter are made in a setting other than the therapeutic setting. Where multiple measures were available for a single outcome (as was sometimes the case for laboratory tasks), the measure most frequently reported across included trials and/or that which was judged to tap the core of the construct was selected. Sensitivity analyses were conducted including only trials meeting the following criteria: use of active/sham control; use of working memory training; use of training targeting more than 1 neuropsychological domain (termed here “multiple process training”); and use of no/low medication (i.e., with <30% of participants receiving medications). We also performed an additional sensitivity analysis excluding the study by Gray et al., in which all participants had a diagnosis of ADHD plus coexisting intellectual disability. Publication bias was assessed with funnel plots and Egger’s tests. Finally, we also conducted a meta-regression analysis, using the metareg command in STATA, to assess the relationship between age and SMD for most proximal and probably blinded assessments of ADHD core symptoms. This analysis was conducted to establish whether the efficacy of cognitive training varied across age, a finding that could be of clinical significance.

12Fifteen trials (reported in 16 papers) met entry criteria (see Table S1, available online). Studies not included in the meta-analysis are listed (with reasons for their exclusion) in Table S2 (available online). Figure 1 reports the trial selection flowchart. Table 1 gives information about retained trials. Results of all analyses are summarized in Table 2. Six trials were on working memory training, 4 on attention training, 2 combined attention and working memory training, 2 inhibition and working memory training, and 1 trial provided a general executive function training covering working memory, inhibition, and cognitive flexibility. All training schedules had an “adaptive” component, that is, task difficulty was increased across sessions to track performance improvement. Eight trials had an active control condition. Six trials were implemented at home, 5 at school, 2 at either school or home, 1 trial in the clinic, and 1 at the welfare service/children center, home or laboratory. Five trials had no/low medication levels. Figure S1 (available online) depicts the graphic output for the risk of bias assessment. Risk of bias was generally low or unclear. No trials were scored as “high risk” with regard to “random sequence generation,” “allocation concealment,” and “incomplete outcome data,” and only 3 and 4 trials scored high for “blinding of participants/personnel” and “blinding of outcome assessment,” respectively (the rating of each study is available upon request).

13ADHD symptoms (total score or inattention or hyperactivity/impulsivity separately) were an outcome in up to 14 trials. Probably blinded measures were available in up to 11 trials (Table 2).

14When most proximal assessments were the outcome, there was a moderate but significant effect on total ADHD and inattention symptoms but no effect on hyperactivity/impulsivity (Figure 2; SMD and CI data for all outcomes are presented in Table 2). In sensitivity analyses (Figures S2 and S3, available online), considering only trials with an active control, the effects were no longer statistically significant for any ADHD core symptoms outcomes. There was no effect of working memory training when implemented on its own (Figures S2 and S3, available online). In contrast, multi-process training approaches (i.e., approaches targeting more than 1 neuropsychological domain) gave a large effect size for total ADHD symptoms (Figure S2, available online). Between-study heterogeneity of effect sizes was high and significant for total ADHD and inattention symptoms.

15When analyses were restricted to probably blinded measures (Figure 2), in general, effect sizes were reduced with small and statistically marginal effects for all ADHD outcomes. In a sensitivity analysis (Figure S4, available online), effect sizes dropped further to nonsignificant levels when only trials with an active control arm were included. There were insufficient studies (n < 5) for an analysis of probably blinded measures in multi-component training trials, as well as for a number of other sensitivity analyses.

16When analysis was restricted to no/low medication trials, effects on total ADHD symptoms were not significant for either most proximal or, when available, probably blinded assessments in any ADHD core symptom–related outcome (Table 2).

17Eight trials included laboratory measures of verbal, and 5 trials visual working memory (Table 2). There was a large and significant effect of cognitive training on both components (Figure 3), which was maintained in sensitivity analyses considering trials with active controls only or working memory training trials only (Figure S5, available online; sensitivity analyses were not performed for visual working memory because of an insufficient number of trials). The number of trials using multi-component training and no/low medication trials was insufficient to perform sensitivity analyses. There were no significant effects of training on laboratory tests of inhibition (6 trials) or attention (7 trials) (Figure 3). Six trials included most proximal ratings of executive functioning using the BRIEF rating scale (Figure S6, available online). These demonstrated a small-to-moderate, significant SMD. There was an insufficient number of trials with ratings of executive functioning to perform planned sensitivity analyses.

18Five trials included standardized measures of reading and 5 of arithmetic. There were no significant effects in either domain (Figure 3). There was an insufficient number of trials to perform planned sensitivity analyses.

19Funnel plots and results of Egger’s test are reported in Supplement 2 (available online). For both meta-analyses of ADHD symptoms scored by most proximal and probably blinded raters, the test failed to reach the p < 0.05 level, suggesting no significant publication bias.

20For most proximal or probably blinded assessments of ADHD core symptoms, there was no significant effect of age on SMD (see Supplement 3, available online).

21The main results considering most proximal assessment of ADHD core symptoms were substantially unchanged, as reported in Figure S7 (available online). As this study was not included in “probably blinded” analyses, no sensitivity analysis was conducted considering probably blinded assessment.

22There are 2 perspectives on cognitive training in ADHD. From 1 perspective, cognitive training is a front-line ADHD treatment: this is based on the hypothesis that because causal pathways to disorder are mediated by neuropsychological deficits, strengthening deficient neuropsychological functions should reduce ADHD symptoms and associated impairment. From the second perspective, it is perceived as an adjunctive treatment that reduces impairment associated with neuropsychological deficits commonly seen in children with ADHD, independent of any effects on core ADHD symptoms itself. The current meta-analysis, including an additional 10 RCTs compared to the previous study by Sonuga-Barke et al., provided little support for cognitive training as a front-line ADHD treatment. There were statistically significant effects on ADHD symptoms when considering raters most proximal to treatment delivery, especially for symptoms of inattention. However, these effects were reduced substantially when analyses were limited to trials with an active control arm or where assessors were probably blind to treatment allocation. The evidence was somewhat stronger for the benefits of cognitive training as an adjunctive treatment aimed at reducing neuropsychological impairment. There were large and highly significant improvements on objective tests of both visual and verbal working memory, although there were no effects on inhibition or inattention. Furthermore, the effects of cognitive training on working memory did not extend to the academic outcomes explored.

23The substantial drop in SMDs between most proximal and probably blinded analyses for ADHD symptoms is similar to the pattern seen in previous meta-analyses of nonpharmacological treatments using the EAGG protocol (e.g., behavioral intervention; neurofeedback). This is probably caused by the inflation of effect size estimates that inevitably occurs when one relies on raters who are both likely to be aware of treatment allocation and heavily invested in the delivery and outcome of treatment. It is also possible that probably blinded and most proximal assessments differed in some way that reduced the sensitivity of the former to treatment-related change. However, the same measurement approaches were used for each (some parent, some teacher, and some direct observation measures). Another possibility is that most proximal assessments accurately captured treatment effects established in the therapeutic setting but that these effects did not generalize to the settings in which probably blinded assessments were made. However, in a substantial minority of trials (Table 1), especially those with an active control arm, probably blinded measures were collected in the treatment setting, and the effects for these trials were no larger than those for trials in which they were collected in a different setting.

24The trials included in the meta-analysis used a wide range of training approaches targeting different neuropsychological processes. There was a sufficient number of trials to look at 2 classes of intervention individually, which was not possible in the previous meta-analysis by Sonuga-Barke et al.: namely, training of working memory only, and training focusing on multiple neuropsychological domains. The results for trials implementing working memory training only departed in a striking way from the most proximal/probably blinded pattern described above. Effects on ADHD were negligible even considering most proximal measures. This suggests that this form of training, which has been widely promoted for use with patients with ADHD (as discussed by Rapport et al.,), has little or no efficacy for core ADHD symptoms. On the other hand, the SMD for most proximal assessment of ADHD symptoms was substantially larger for trials based on training targeting multiple domains than for all studies as a whole. Unfortunately, there was an insufficient number of trials (n = 4) with probably blinded measures to corroborate these effects using independent sources. The superiority of these approaches may be due to the typically greater number of training sessions in multi- compared to single-component approaches (in our analysis, an average of 9 weeks compared to 6 weeks, respectively). However, the finding opens up the interesting possibility that multi-component training models may be more successful for ADHD, given the complex and heterogeneous nature of the condition. Because children with ADHD differ from one another in their neuropsychological profile, and because children may be affected by more than 1 deficit, multi-component training may be used to target a series of neuropsychological domains that may be more important than working memory alone in the pathophysiology of ADHD symptoms. The development and evaluation of multi-component training models should be a future priority.

25The effects on neuropsychological outcomes were restricted to working memory, which were substantial, with no effects on inhibitory or attentional control. There were significant effects on parents’ ratings of executive function, but these could not be corroborated by independent blinded evidence. All 6 trials that included a working memory outcome were working memory training trials. Therefore, although these trials produced “near transfer” of training effects to untrained working memory measures, there was no evidence of “far transfer” to other neuropsychological processes. Crucially, there was also no evidence that these effects generalized to important areas of everyday functioning, which themselves are influenced by working memory ability, such as reading and arithmetic. This finding may be relevant in clinical practice. Indeed, parents may currently favor cognitive training with the hope that they can improve academic performance. Our results show that this is not supported by empirical evidence.

26The success of working memory training in improving working memory performance draws into even sharper relief its failure to improve ADHD symptoms, suggesting dissociation between neuropsychological functioning and disorder. There are 4 possible explanations for this: first, that working memory deficits do not, in fact, mediate ADHD pathophysiology; second, that although they do mediate the development of ADHD, they have become entrenched and not susceptible to the type of training implemented in trials conducted to date; third, that training as currently implemented targets types of working memory not fundamental to the deficits in ADHD; and fourth, that training produces only peripheral, practice-like effects on working memory, with no profound impact on the brain networks underpinning neuropsychological deficits responsible for ADHD. Whether or not working memory deficits are part of the causal mechanism underpinning ADHD, based on our results, strengthening working memory appears to be neither a necessary nor a sufficient condition for ADHD symptom reduction. In this regard, our findings suggest that choosing substrates that have emerged from experimental research as treatment targets may not necessarily translate into clinical benefits. This possible dissociation between candidate mechanisms of a disorder and clinical targets is important when adopting pathophysiology-based research approaches such as the Research Domain Criteria (RDoC). From a clinical standpoint, developing techniques to extend transfer from the effects on core working memory processes to broader neuropsychological processes and important domains of impairment and/or clinical presentation is the most pressing challenge for the future. The reasons for the lack of effect on inhibitory and attentional control are hard to determine on the basis of the current analysis, given the small number of trials that specifically targeted these domains. Although we might predict that training targeting multiple deficit domains would show effects on these neuropsychological processes, there were insufficient trials with multi-component training and measures of inhibition and/or attention to test this. Approaches focusing on motivational or energetic processes may also be valuable (i.e., training to increase delay of gratification).

27A number of limitations need to be taken into account when interpreting the current analysis. First, there was significant SMD heterogeneity for some analyses (most proximal total ADHD, symptoms of inattention, and visual working memory). This leaves open the possibility that cognitive training may be effective under specific circumstances in individual trials. Given the limited number of trials available, we were unable to identify specific features of positive trials (apart from therapeutic content, working memory training). Second, only a minority of trials (n = 5) reported using intention-to-treat analyses, a situation that may have inflated the effects for some outcomes, as participants who are more difficult to treat or who perceive the treatment as less beneficial may drop out of trials; however, drop-out was relatively low in most trials. Third, despite the recent substantial increase in the number of available cognitive training trials, there was an insufficient number of trials to evaluate training approaches targeting specific neuropsychological constructs other than working memory training. Fourth, there were insufficient trials to run analyses for some important outcomes (e.g., functional impairment, IQ), as well as for sensitivity analyses or analyses restricted to probably blinded measures for a number of outcomes. Fifth, too few trials included long-term outcomes (Table 1) to allow an evaluation of the extent to which effects on clinical symptoms grew over time or effects on neuropsychological processes persisted. Sixth, no trials were restricted to individuals with both ADHD and the specific neuropsychological deficit to be trained. As a consequence, effect sizes for both neuropsychological deficits and ADHD symptoms may have been truncated: in the former case because there would be little room for improvement where no deficit existed; in the latter case because targeting a neuropsychological deficit that was not causing the condition would be unlikely to reduce symptoms of the core condition. Seventh, in the neuropsychological domains, diverse measures from different tasks (still tapping the same domain, however) were combined across studies to allow the calculation of pooled SMD estimates. Eighth, it is important to understand whether initial symptom-related and neuropsychological treatment effects persist over time and generalize to other domains, if they do. There were insufficient trials that examined long-term outcomes to address this issue. Finally, the categorization of studies as “probably blinded,” although carried out according to previously agreed and clear decision rules set out in the protocol, is limited by an inevitable degree of uncertainty because of limitations in the information reported in some trials.

28In summary, the current meta-analysis found limited evidence for the clinical value of cognitive training for children with ADHD outside of the narrow confines of specific targeted neuropsychological processes (i.e., working memory training improved working memory function). Given the evidence for neuropsychological heterogeneity in ADHD, future efforts should be directed at developing protocols to target a broader range of neuropsychological deficits. Furthermore, therapeutic innovation is required to enhance the “far transfer” of specific neuropsychological gains to everyday patterns of functional impairment through more ecologically valid training approaches. Future trials should more consistently include active control arms, a broader range of functional outcomes, and long-term follow-up.