Validity of activity monitors in health and chronic disease: a systematic review

1There is evidence that regular physical activity is associated with a reduced risk of mortality and contributes to the primary and secondary prevention of several chronic diseases. For example, a reduced risk of coronary heart disease, cardiovascular disease, stroke and colon cancer has been reported in more active individuals. In patients with chronic obstructive pulmonary disease (COPD), regular physical activity leads to a lower risk of both COPD related hospital admissions and mortality. Physical activity limitation is a major problem in patients with chronic diseases and needs to be accurately measured if therapies aimed at improving this are to be properly evaluated. A range of devices are available for this purpose but most have been validated in young, healthy subjects and their applicability to older or unwell populations, where movements tend to be slower, is not well established.

2Physical activity is defined as any bodily movement, produced by skeletal muscles, requiring energy expenditure. Daily physical activity can be considered as “the totality of voluntary movement produced by skeletal muscles during everyday functioning”. Estimates of daily physical activity can be obtained by different approaches; questionnaires, energy expenditure measurements and activity monitors. Questionnaires rely on the subject’s recollection of activities and allow categorization of patients by physical activity (very active, active, sedentary and inactive), but may lack the precision needed to detect changes in physical activity on a day to day basis.

3Daily physical activity can be expressed as an overall measure of active energy expenditure, using indirect calorimetry techniques such as doubly labelled water or metabolic carts. Although doubly labelled water is regarded as a criterion method, this technique does not quantify the duration, frequency and intensity of physical activity performed. Metabolic cart systems which measure expired O2 and CO2 however cannot be used over extended periods of time.

4Physical activity can also be monitored directly using physical activity monitors. In general, three classes of activity monitors are being used increasingly in chronic disease populations (e.g. COPD): pedometers, accelerometers and integrated multisensor systems. Pedometers are devices which estimate the number of steps taken through mechanical or digital measurements in only the vertical plane. This is a limited measure of physical activity. Accelerometers detect acceleration in one, two or three directions (uni-, bi- or triaxial accelerometers). These devices allow determination of the quantity and intensity of movements. Integrated multisensor systems combine accelerometry with other sensors that capture body responses to exercise (e.g. heart rate or skin temperature) in an attempt to optimise physical activity assessments.

5With the advancement of technology, the number of activity monitors available to measure physical activity is growing. However, despite these advances, it remains a challenge to assess physical activity in slowly moving patients (such as those with COPD, chronic heart failure and diabetes type II). In these patients small changes in physical activity are likely to be important effects of interventions aimed at enhancing physical activity. Therefore, in order for investigators to interpret the effect of interventions on physical activity, activity monitors that have been properly validated in these patient groups are needed.

6In order to make evidence based statements on the validity of activity monitors, a systematic review was conducted to identify available activity monitors that have been validated in both healthy adults and chronic disease populations.

7Studies meeting the following criteria were included: (1) Population: healthy adults and adults with a diagnosis of chronic disease in whom inactivity is a likely contributor to morbidity or a target for treatment, but whose locomotor function is relatively preserved (COPD, heart failure, diabetes type II, frail elderly, primary pulmonary hypertension, chronic low back pain, fibromyalgia syndrome, obesity). (2) Measurement: any commercially available activity monitor for outdoor activity monitoring from uniaxial to triaxial accelerometers and multisensor devices to tools incorporating spatial information (e.g. GPS) or other information on motion. (3) Study design: studies that evaluated the validity of an activity monitor, i.e. testing an activity monitor against a criterion method, such as indirect calorimetry. Two types of validation studies were included; field validation studies (validation of an activity monitor against doubly labelled water) and laboratory validation studies (validation of an activity monitor using a metabolic cart or metabolic chamber and/or manual step-counting or video observation). (4) Clinical trials using activity monitoring as an outcome and which might contain a reference to a validation paper were included for hand-searching. (5) A search window between 1st of January 2000 until 1st of March 2012 was selected in order to capture sensors in contemporary use. This approach still allowed for the identification of older validation studies (published before 2000) of devices in current use in clinical trials. Main exclusion criteria were 1) studies in children (subjects younger than 18 years), 2) studies in subjects with abnormal biomechanical movement patterns (e.g. cerebral palsy, lower limb amputation), 3) studies only investigating the number of steps using pedometers because of the inaccuracy in measurement of total energy expenditure and lack of ability to measure physical activity patterns.

8No language restrictions were used; any non-English studies retrieved through the literature search were translated to determine their appropriateness for inclusion.

9Eligible studies were identified by searching the following databases: MEDLINE, EMBASE and CINAHL. A librarian was consulted prior to initiating the search in order to identify appropriate search terms to describe the population (from healthy adults to patients with chronic disease), physical activity and activity monitoring. A combination of MeSH terms (MEDLINE), Emtree terms (Embase) and Cinahl headings (Cinahl) with free text words (all databases) were used (see Additional file 1 for detailed information). Refworks (www.refworks.com) was used to store and share all papers and to collect all the information of title and abstract screening, full text assessment and the hand-searching process. Each review team consisted of 3 reviewers who independently screened the titles and abstracts of the retrieved articles. Each abstract was labelled as ‘A) excluded papers‘, ‘B) order for full text assessment‘or ‘C) hand-search for references only’, i.e. clinical trials which may have a reference to an older validation study. After independently reviewing the articles for inclusion, the reviewers compared their labels to ensure consensus. Once agreement had been reached, a full text copy of each article that met the inclusion criteria was obtained (Label B). Thereafter, the same review teams looked at the full texts of the potential validation papers in detail and decided in consensus, whether the articles were indeed suitable validation papers for data extraction. Subsequently, hand-searching of the clinical trials using an activity monitor outcome which might contain a reference to a validation paper (Label C), was performed by three independent reviewers. After independently reviewing these full texts, validation papers were identified which met the inclusion criteria for full text assessment. Again, the reviewers compared their decisions to ensure consensus. Data of all included validation papers were extracted into predefined prepared Excel tables.

10For the field studies, correlation coefficients between total and active energy expenditure from activity monitor (TEEAM and AEEAM respectively) and total and active energy expenditure measured with doubly labelled water (TEEDLW and AEEDLW respectively) were extracted. The percentage mean differences (ΔTEE and ΔAEE) with 95% confidence intervals were obtained from the reports to assess agreement between energy expenditure estimates from the activity monitor (TEEAM and AEEAM) versus energy expenditure measures from doubly labelled water (TEEDLW and AEEDLW). For the laboratory studies, correlation coefficients between activity monitor outcome and EE measured by metabolic cart/chamber were extracted. A sub-analysis included to compare correlation coefficients derived from walking based protocols to correlation coefficients derived from protocols based on activities of daily living. Agreement between energy expenditure outcomes from the activity monitor versus criterion method (indirect calorimetry) were extracted by the mean difference at different treadmill walking speeds; slow walk (<3.2 km/hr or 1 mph), intermediate speed walk (3.2-6.4 km/hr or 2–4 mph), fast walk (6.5-8.05 km/hr or 4–5 mph) and running (8.06-11 km/hr or 5–7 mph). Accuracy of steps measured by activity monitoring was expressed as the percentage mean difference between steps measured by an activity monitor versus actual steps measured by the criterion method (video observation and/or manual step counting).

11Descriptive statistics were used to report information about type of activity monitor, activity monitor outcomes and studied population. Papers were separated by type of validation, ‘field validation papers’ (validation of an activity monitor against indirect calorimetry, using the doubly labelled water technique) and ‘lab validation papers’ (validation of an activity monitor against indirect calorimetry, using a metabolic cart, metabolic chamber or direct observation).

12We also analysed the results separately per type of device (uni-, bi-, triaxial and multisensor devices). We performed (DerSimonian and Laird) random-effects meta-analyses to pool the correlation coefficients and mean differences across studies and expressed heterogeneity by the I2 statistic, which estimates the percentage of total variation between studies that is due to heterogeneity rather than chance. I2 is calculated from basic results obtained from a typical meta-analysis as I2 = 100% x (Q-df)/Q, where Q is Cochran’s heterogeneity statistic and df the degrees of freedom. Negative values of I2 are put equal to zero so that I2 lies between 0% en 100% with larger values showing larger heterogeneity. We used the Fisher r to z-transformation in order to pool normally distributed data (z scores) rather than the skewed distribution of Pearson correlation coefficients. We back transformed the pooled z-scores to correlation coefficients for easier interpretation.

13We used random-effects linear regression models (meta-regression analyses) with the studies’ results as the dependent variable (and considering each studies’ standard error) to compare the type of devices (covariate) and to assess the type of population (covariate) as a potential explanation for heterogeneity. For those few studies where no measures of variability were reported we imputed the median standard deviations of those studies where the standard deviation was available. We did not perform meta-analyses for the laboratory studies where none of the studies provided standard deviations for ΔTEE and ΔAEE but presented the point estimates as graphs. Coefficient of variation for TEEDLW and AEEDLW was calculated per study population to investigate whether the degree of variation in TEEDLW and AEEDLW affected the correlation coefficients and/or mean differences, (i.e. higher correlations/mean differences in populations with larger variation in TEE and/or AEE).

14The systematic literature search resulted in a total of 2875 abstracts which were scrutinised by four review teams across Europe. Figure 1 represents the different processes used in the systematic review.

15Forty monitors were tested in validation studies; 12 uniaxial, 3 biaxial, 16 triaxial accelerometers and 9 multisensor devices. Fifty-five percent of activity monitors (22/40) were used only in lab validation studies, 10% (4/40) only in field validation studies and 35% (14/40) in both a lab as well as a field validation study. An overview of the different activity monitors is shown in Tables 1, 2, 3 and 4.

16The most frequently available outcomes present in validated activity monitors are (total and/or active) energy expenditure (70%, 28/40), steps (38%, 15/40) and different levels of physical activity intensity (38%, 15/40). The majority of the validation studies (118/134, 88%) were performed in healthy adults. Few studies (16/134, 12%) were performed in chronic disease populations; obesity (n = 4), chronic obstructive pulmonary disease (n = 5), chronic heart failure (n = 1), chronic organ failure (n = 1), chronic low back pain (n = 1), fibromyalgia syndrome (n = 1), peripheral arterial disease (n = 1), diabetes mellitus type II (n = 1) and a general chronic disease population (cardiac, obese or knee arthritis, n = 1).

17Individual correlation coefficients, with converted Fisher z-scores, for total energy expenditure (TEE) between TEEAM and TEEDLW are presented in Figure 2.

18Variability of study populations’ TEEDLW was relatively small; coefficient of variation (CV) ranged from 0.11 to 0.29. Pooled r in uniaxial devices (r = 0.52 (95%CI, 0.29 to 0.70)) was significantly lower compared to multisensor devices (r = 0.84 (95%CI, 0.78 to 0.88), p<0.001) but not to triaxial devices (r = 0.61 (95%CI, 0.45 to 0.73, p = 0.37)). Because of the relatively large difference in accuracy between the uniaxial, the triaxial and multisensor devices 53% of the between–study heterogeneity was accounted for by type of device in meta-regression analyses.

19ΔTEE (TEEAM – TEEDLW) was less accurate in uniaxial compared to triaxial accelerometers and multisensor devices (−12.07 (95%CI, -18.28 to −5.85) % in uniaxial versus −6.85 (95%CI, -18.20 to 4.49) % in triaxial (p = 0.39 for comparison against uniaxial devices) and −3.64 (95%CI, -8.97 to 1.70) % in multisensor devices, p = 0.03 for comparison against uniaxial devices, Figure 3). ΔTEE were smaller in studies with chronic disease populations than in studies with healthy populations (−9% (95%CI −19 to 1)) but the difference did not reach statistical significance (p = 0.09).

20Correlations for active energy expenditure (AEE) between AEEAM and AEEDLW were higher in triaxial (0.59 (95%CI, 0.45 to 0.70)) and multisensor devices (0.54 (95%CI, 0.39 to 0.65)) compared to uniaxial (0.39 (95%CI, 0.16 to 0.58)) devices, p = 0.12 for triaxial and p = 0.32 for multisensor against uniaxial devices) (Figure 4) Types of devices accounted for only 12% of the between-study heterogeneity in the meta-regression analysis. All monitors underestimated AEE (ΔAEE (AEEAM – AEEDLW) -24.22 (95%CI, -62.05 to −13.61) % in uniaxial, -21.01 (95%CI, -41.92 to −0.11) % in triaxial and −24.35 (95%CI, -45.28 to −3.42) % in multisensor devices. No significant differences were found between devices (Figure 5). But ΔAEE were statistically significantly smaller in studies with chronic disease populations than in studies with healthy populations (−44%, 95%CI −73 to −13, p = 0.006).

21For correlation analysis, TEE and AEE, as determined from indirect calorimetry, were used as criterion outcomes (in 89% and 11% of the studies, respectively) against different outcomes of the activity monitor (activity counts (37%), vector magnitude units (7%), total energy expenditure (48%), active energy expenditure (2%) or monitor-specific activity scores (6%).

22Pooled correlation coefficients between indirect calorimetry (TEEIC) and activity monitor outcome were lower in uniaxial (0.80 (95%CI, 0.75 to 0.84)) compared to multisensor devices (0.85 (95%CI, 0.72 to 0.92)) but the difference did not reach statistical significance (p = 0.43) No differences were found with biaxial (0.73 (95%CI, 0.33 to 0.91), p = 0.50) and triaxial (0.84 (95%CI, 0.78 to 0.89), p = 0.28) devices, either (Figure 6).

23Correlation coefficients between TEEIC and activity monitor outcome were higher when tested using laboratory protocols based on walking activities (overall pooled r = 0.84 (95%CI, 0.79 to 0.87), no significant differences between types of devices, Figure 7) compared to protocols using activities of daily living involving the upper and lower limbs (overall pooled r = 0.75 (95%CI, 0.68 to 0.81, no significant differences between types of devices), Figure 8).

24There was evidence of heterogeneity of results across all analyses (overall I2 ranged from 84.6% (Figure 7) to 85.9% (Figure 8)). Again, the results did not differ for chronic disease and healthy populations in any of the analyses on laboratory validation studies.

25Mean differences between TEEAM and TEEIC at different treadmill walking speeds are presented in Figures 9, 10, 11 and 12. TEE was underestimated during slow walking speed in 69% of studies (n = 16/23), whereas in only 37% of studies (n = 15/40) during intermediate walking speed, 30% of studies (n = 10/33) during fast walking speed and 37% of studies (n = 7/19) during running reported underestimation of TEE. Underestimations in the slow walking group were relatively larger.

26All accelerometers underestimate steps during slow walking; from 0.94 to 60% underestimation. One uniaxial device (activPAL), mounted on the thigh, showed a high accuracy in measuring steps during slow walking with only 0.94% overestimation. More accurate estimates of steps were reported at higher speeds; from 13% under to 2% overestimation during intermediate walking speed (except one study with 35% underestimation using SenseWear Armband), and from 0.18 to 4.3% overestimation during fast walking (Figure 13).

27This systematic review of the literature identified forty activity monitors (12 uniaxial, 3 biaxial, 16 triaxial and 9 multisensor devices) that had been validated against indirect calorimetry (doubly labelled water, metabolic cart and/or metabolic chamber) in healthy adults (88% of studies) or adults with chronic disease (12% of studies).

28Field and laboratory validation studies had highly heterogeneous results which could partly be explained by the type of activity monitor and the activity monitor outcome. These factors need consideration when a validation study is evaluated.

29First, selecting the type of activity monitor is important. Pedometers are limited in their ability to detect certain physical activity patterns which might occur in chronic disease populations (for example, an unstable gait profile or lack of intensity of physical activity). Accelerometers can overcome this. Multi-axial accelerometers have the ability to measure accelerations in different orientations, which provides information about the total amount, intensity and duration of daily physical activity. Some multisensor devices, which combine physiological parameters with accelerometry, are available to assess both body posture and body movement. An additional promising class of monitors integrate positioning systems (Global Positioning System (GPS) and Bluetooth® systems for outdoor and indoor activities respectively) with accelerometry and other sensors. However, to date, these have been used infrequently in patients with chronic disease. Based on this systematic review, heterogeneity among studies was significantly explained by the types of devices, although no statistical significance was reached between different types of devices.

30A second factor to take into consideration is the activity monitor outcome. When measuring TEE in field validation studies (doubly labelled water), high correlations with the TEE estimate of the activity monitor were found in most activity monitors. These correlations are, however, to a large extent driven by patient characteristics (i.e. body weight, age, height) which is an important predictor of TEE. Consequently, the comparison of TEE estimated from activity monitors, with TEE measured with indirect calorimetry or doubly labelled water is not necessarily a proof of validation. In a field setting it has been reported that only 19% of the TEE is accounted for by physical activity in both healthy subjects and in patients with coronary heart disease.

31Another factor that needs to be considered is the study population. Most of the study populations (88%) were healthy adults (from young healthy adults to healthy elderly). Only 12% of validation studies were performed in patients with chronic diseases (COPD, chronic heart failure, chronic organ failure, diabetes mellitus type II, obesity, peripheral arterial disease chronic low back pain and fibromyalgia syndrome). These patients walk more slowly than healthy subjects, which is reflected, for example, by a reduced six minute walking distance. This review, as well as original research, suggests that most monitors are less accurate at lower walking speeds. These findings are consistent with a systematic review of pedometers which found evidence of reduced accuracy during slow walking. Hence, there is a need to perform validation studies specifically in chronic disease populations.

32When measuring TEE in lab validation studies by assessment of oxygen consumption, higher correlations were reported for walking activities compared to other daily life activities which implies that the walking component of physical activity is better detected than other activities of daily living.

33Most activity monitors use prediction equations to calculate energy expenditure from the activity signals. This is helpful to validate monitors against indirect calorimetry, but, given the inherent inaccuracy of these estimates and fundamental differences between the different prediction equations (some of which are proprietary to particular device manufacturers), perhaps greater weight should be given to direct monitor outputs (steps, activity counts, VMU, etc.) and their relation to activity energy expenditure (AEE), rather than the ability of a monitor to estimate energy expenditure precisely. It is very unlikely that an activity monitor will be able to capture accurately all the factors affecting energy expenditure (i.e. movement efficiency, resting metabolism, distribution of fat-free mass and fat mass). In patients with COPD, for example, Baarends et al. showed that non-resting energy expenditure (TEE-REE) was elevated in COPD compared to healthy controls. Since it is generally accepted now that these patients are less active than healthy controls, it is clear that patients expend more energy than controls to achieve the same movements. It would be unrealistic to expect an activity monitor to pick this up. Hence, the lack of accuracy against energy expenditure does not render activity monitors invalid tools to assess physical activity in patients over time (for which precision is more important) or to capture the physical activity level of a patient (for which validity, represented by the correlation with true energy expenditure is more important than absolute accuracy). The acceptable correlations between VO2 and activity monitor outputs in triaxial and multisensor devices are therefore encouraging for the use of monitors to assess physical activity in an adult population. With specific validation studies, these findings can possibly be extrapolated to elderly and patients with chronic diseases.

34The current systematic review may also help researchers to decide on appropriate activity monitor outcomes. Combination of the three most frequently available outcomes (TEE/AEE, steps and different levels of physical activity intensity), which is likely to provide a comprehensive insight in overall physical activity of a patient, is available in 3 uniaxial (Actigraph 7164/GT1M, Kenz Lifecorder EX and Polar Activity Watch 200), 1 biaxial (Biotrainer Pro), 3 triaxial (Dynaport Minimod, Actical and Actigraph GT3X) and 2 multisensor activity monitors (SenseWear Armband and multisensor board). Some general considerations can also be taken into account when selecting an activity monitor in clinical trials such as the type of monitoring (e.g. daily physical activity), size and scope of the study, usability of the monitor and cost.

35A point of difficulty in collecting, analysing and interpreting the data was the wide range of statistical approaches used in the original papers. Indeed, we had to compute the standard deviation of the mean difference (between EEAM and EEIC) because some field validation studies didn’t report this.

36Correlation analysis but also Bland and Altman analysis were the two main statistical approaches used in validation studies and were used for data extraction. A systematic review of the statistical methods used to validate physical activity questionnaires revealed similar findings, with the majority of the studies using correlation analysis compared to Bland and Altman analysis. Correlation analyses are a common evaluation approach and allow statements on validity, whereas agreement between activity monitor and criterion method (indirect calorimetry) with Bland and Altman plots are preferred when the aim is to identify systematic bias in measures. Since not all activity monitors have the possibility to estimate total and/or active energy expenditure, this type of analysis is not uniformly applicable. Multiple regression analysis with TEE/AEE as the dependent variable is a correct technique to tackle this. Consistent statistical guidelines for reporting the validity of an activity monitor would be helpful.

37Validation studies of activity monitors are highly heterogeneous, and this is partly explained by the type of activity monitor and the activity monitor outcome. Since activity monitors are less accurate at slow walking speeds and information about validated activity monitors in chronic disease populations is lacking, proper validation studies in these populations are needed prior to their inclusion in clinical trials.

38AEE, Active energy expenditure; AEEAM, Estimation of active energy expenditure by an activity monitor; AEEDLW, Measurement of active energy expenditure by doubly labelled water; AEEIC, Measurement of active energy expenditure by indirect calorimetry; AM, Activity monitor; COPD, Chronic obstructive pulmonary disease; DLW, Doubly labelled water; GPS, Global positioning system; IC, Indirect calorimetry; TEE, Total energy expenditure; TEEAM, Estimation of total energy expenditure by an activity monitor; TEEDLW, Measurement of total energy expenditure by doubly labelled water; TEEIC, Measurement of total energy expenditure by indirect calorimetry.

39The authors declare not having financial or non-financial competing interests.

40All authors contributed to the different processes in the systematic review (title and abstract screening, full text assessment and data extraction). HVR worked out an appropriate search term strategy in different databases. HVR and SG have been involved in analysing the data and drafting the manuscript which was revised by all the other authors. MP performed the statistical analysis. All authors read and approved the final manuscript.

41This systematic review is part of the PROactive project (www.proactivecopd.com) which aims to develop a Patient Reported Outcome (PRO) tool capturing physical activity in COPD in close harmony with a valid activity monitor.