No retraction, correction or concern found when last checked on Oct 16, 2026.

The heterogeneity statistic I2 can be biased in small meta-analyses

Paul T von Hippel

BMC Medical Research Methodology · 2015

Abstract

Background. Estimated effects vary across studies, partly because of random sampling error and partly because of heterogeneity. In meta-analysis, the fraction of variance that is due to heterogeneity is estimated by the statistic I2. We calculate the bias of I2, focusing on the situation where the number of studies in the meta-analysis is small. Small meta-analyses are common; in the Cochrane Library, the median number of studies per meta-analysis is 7 or fewer.

Methods. We use Mathematica software to calculate the expectation and bias of I2.

Results. I2 has a substantial bias when the number of studies is small. The bias is positive when the true fraction of heterogeneity is small, but the bias is typically negative when the true fraction of heterogeneity is large. For example, with 7 studies and no true heterogeneity, I2 will overestimate heterogeneity by an average of 12 percentage points, but with 7 studies and 80 percent true heterogeneity, I2 can underestimate heterogeneity by an average of 28 percentage points. Biases of 12–28 percentage points are not trivial when one considers that, in the Cochrane Library, the median I2 estimate is 21 percent.

Conclusions. The point estimate I2 should be interpreted cautiously when a meta-analysis has few studies. In small meta-analyses, confidence intervals should supplement or replace the biased point estimate I2.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 26

1.
Melsen WG, Bootsma MCJ, Rovers MM, Bonten MJM. The effects of clinical and statistical heterogeneity on the predictive values of results from meta-analyses. Clin Microbiol Infect. 2014;20(2):123-9.
Not in our library10.1111/1469-0691.12494
2.
Ioannidis JPA, Patsopoulos NA, Evangelou E. Uncertainty in heterogeneity estimates in meta-analyses. BMJ. 2007;335(7626):914-6.
Not in our library10.1136/bmj.39343.408449.80
3.
Characteristics of meta-analyses and their component studies in the Cochrane Database of Systematic Reviews: a cross-sectional, descriptive analysis · 2011
No notice
4.
Cochran WG. The combination of estimates from different experiments. Biometrics. 1954;10:101-29.
Not in our library10.2307/3001666
5.
Hardy RJ, Thompson SG. Detecting and describing heterogeneity in meta-analysis. Stat Med. 1998;17(8):841-56.
Not in our library10.1002/(sici)1097-0258(19980430)17:8<841::aid-sim781>3.0.co;2-d
6.
Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21(11):1539-58.
Not in our library10.1002/sim.1186
7.
Koedel C. An empirical analysis of teacher spillover effects in secondary school. Econ Educ Rev. 2009;28(6):682-92.
Not in our library10.1016/j.econedurev.2009.02.003
8.
Koedel C, Parsons E, Podgursky M, Ehle M. Teacher preparation programs and teacher quality: are there real differences across programs? 2012.
Not in our library
9.
von Hippel PT, Osborne C, Lincove A, Bellows L, Mills N. The challenges of seeking exceptional teacher preparation programs among many noisy estimates. 2014.
Not in our library
10.
Kivimäki M, Batty GD, Ferrie JE, Kawachi I. Cumulative meta-analysis of job strain and CHD. Epidemiology. 2014;25(3):464-5.
Not in our library10.1097/ede.0000000000000087
11.
Aune D, Saugstad OD, Henriksen T, Tonstad S. Physical activity and the risk of preeclampsia: a systematic review and meta-analysis. Epidemiology. 2014;25(3):331-43.
Not in our library10.1097/ede.0000000000000036
12.
Crippa A, Discacciati A, Larsson SC, Wolk A, Orsini N. Coffee consumption and mortality from all causes, cardiovascular disease, and cancer: a dose–response meta-analysis. Am J Epidemiol. 2014;180(8):763-75.
Not in our library10.1093/aje/kwu194
13.
Kim Y, Je Y. Dietary fiber intake and total mortality: a meta-analysis of prospective cohort studies. Am J Epidemiol. 2014;180(6):565-73.
Not in our library10.1093/aje/kwu174
14.
Cochrane Collaborative, “Cochrane Library”, 2015. [Online]. Available: http://www.cochranelibrary.com/. [Accessed: 25-Feb-2015].
Not in our library
15.
Hedges LV, Vevea JL. Fixed- and random-effects models in meta-analysis. Psychol Methods. 1998;3(4):486-504.
Not in our library10.1037/1082-989x.3.4.486
16.
A Re-Evaluation of Random-Effects Meta-Analysis · 2008
No notice
17.
Undue reliance on I 2 in assessing heterogeneity may mislead · 2008
No notice
18.
Biggerstaff BJ, Jackson D. The exact distribution of Cochran’s heterogeneity statistic in one-way random effects meta-analysis. Stat Med. 2008;27(29):6093-110.
Not in our library10.1002/sim.3428
19.
Hedges LV, Pigott TD. The power of statistical tests in meta-analysis. Psychol Methods. 2001;6(3):203-17.
Not in our library10.1037/1082-989x.6.3.203
20.
Engels EA, Schmid CH, Terrin N, Olkin I, Lau J. Heterogeneity and statistical significance in meta-analysis: an empirical study of 125 meta-analyses. Stat Med. 2000;19(13):1707-28.
Not in our library10.1002/1097-0258(20000715)19:13<1707::aid-sim491>3.0.co;2-p
21.
Bock ME, Judge GG, Yancey TA. A simple form for the inverse moments of non-central χ2 andF random variables and certain confluent hypergeometric functions. J Econometrics. 1984;25(1–2):217-34.
Not in our library10.1016/0304-4076(84)90048-4
22.
Turner RM, Davey J, Clarke MJ, Thompson SG, Higgins JP. Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane Database of Systematic Reviews. Int J Epidemiol. 2012;41(3):818-27.
Not in our library10.1093/ije/dys041
23.
Viechtbauer W. Bias and efficiency of meta-analytic variance estimators in the random-effects model. J Educ Behav Stat. 2005;30(3):261-93.
Not in our library10.3102/10769986030003261
24.
Chung Y, Rabe-Hesketh S, Choi I-H. Avoiding zero between-study variance estimates in random-effects meta-analysis. Stat Med. 2013;32(23):4071-89.
Not in our library10.1002/sim.5821
25.
Hartung J, Knapp G. On confidence intervals for the among-group variance in the one-way random effects model with unequal error variances. J Stat Plann Infer. 2005;127(1–2):157-77.
Not in our library10.1016/j.jspi.2003.09.032
26.
Ray KK, Seshasai SRK, Erqou S, Sever P, Jukema JW, Ford I, et al. Statins and all-cause mortality in high-risk primary prevention: a meta-analysis of 11 randomized controlled trials involving 65,229 participants. Arch Intern Med. 2010;170(12):1024-31.
Not in our library10.1001/archinternmed.2010.182

Cited by papers we hold 7

Estimating the global prevalence of chronic obstructive pulmonary disease (COPD): a systematic review and meta-analysis · 2024
No notice
Health effects associated with smoking: a Burden of Proof study · 2022
No notice
Multi-omics signatures of the human early life exposome · 2022
No notice
Systematic review and meta-analysis: effects of maternal separation on anxiety-like behavior in rodents · 2020
No notice
Prognostic significance of blood inflammatory biomarkers NLR, PLR, and LMR in cancer—A protocol for systematic review and meta-analysis · 2019
No notice
Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis · 2019
No notice
Maternal influenza and birth outcomes: systematic review of comparative studies · 2016
No notice

Reads

Read by people on this page

None yet.

Checked or read by AI through naulon

None yet.

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.

Are you an author?

Is this your paper?

Claim it to add your note and be paid when agents cite it. Free.