Skip to content
naulon
Get started

No retraction, correction or concern found when last checked on Oct 17, 2026.

Hartung-Knapp-Sidik-Jonkman approach and its modification for random-effects meta-analysis with few studies

Christian Röver, Guido Knapp, Tim Friede

BMC Medical Research Methodology · 2015

doi:10.1186/s12874-015-0091-1

Abstract

Background. Random-effects meta-analysis is commonly performed by first deriving an estimate of the between-study variation, the heterogeneity, and subsequently using this as the basis for combining results, i.e., for estimating the effect, the figure of primary interest. The heterogeneity variance estimate however is commonly associated with substantial uncertainty, especially in contexts where there are only few studies available, such as in small populations and rare diseases.

Methods. Confidence intervals and tests for the effect may be constructed via a simple normal approximation, or via a Student-t distribution, using the Hartung-Knapp-Sidik-Jonkman (HKSJ) approach, which additionally uses a refined estimator of variance of the effect estimator. The modified Knapp-Hartung method (mKH) applies an ad hoc correction and has been proposed to prevent counterintuitive effects and to yield more conservative inference. We performed a simulation study to investigate the behaviour of the standard HKSJ and modified mKH procedures in a range of circumstances, with a focus on the common case of meta-analysis based on only a few studies.

Results. The standard HKSJ procedure works well when the treatment effect estimates to be combined are of comparable precision, but nominal error levels are exceeded when standard errors vary considerably between studies (e.g. due to variations in study size). Application of the modification on the other hand yields more conservative results with error rates closer to the nominal level. Differences are most pronounced in the common case of few studies of varying size or precision.

Conclusions. Use of the modified mKH procedure is recommended, especially when only a few studies contribute to the meta-analysis and the involved studies’ precisions (standard errors) vary.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

Read at PubMed Central

References 44

  1. 1.

    Hedges LV, Olkin I. Statistical Methods for Meta-analysis. 1985.

    Not in our library

  2. 2.

    Hartung J, Knapp G, Sinha BK. Statistical Meta-analysis with Applications. 2008.

    Not in our library

  3. 3.

    Viechtbauer W. Bias and efficiency of meta-analytic variance estimators in the random-effects model. J Educ Behav Stat. 2005;30(3):261-93.

    Not in our library10.3102/10769986030003261

  4. 4.

    Sidik K, Jonkman JN. A comparison of heterogeneity variance estimators in combining results of studies. Stat Med. 2007;26(9):1964-81.

    Not in our library10.1002/sim.2688

  5. 5.

    Panityakul T, Bumrungsup C, Knapp G. On estimating heterogeneity in random-effects meta-regression: A comparative study. J Stat Theory and Appl. 2013;12(3):253-65.

    Not in our library

  6. 6.
    Methods to estimate the between‐study variance and its uncertainty in meta‐analysis · 2015
    No notice
  7. 7.

    Follmann DA, Proschan MA. Valid inference in random effects meta-analysis. Biometrics. 1999;55(3):732-7.

    Not in our library10.1111/j.0006-341x.1999.00732.x

  8. 8.

    Hartung J, Knapp G. On tests of the overall treatment effect in meta-analysis with normally distributed responses. Stat Med. 2001;20(12):1771-82.

    Not in our library10.1002/sim.791

  9. 9.

    Hartung J, Knapp G. A refined method for the meta-analysis of controlled clinical trials with binary outcome. Stat Med. 2001;20(24):3875-89.

    Not in our library10.1002/sim.1009

  10. 10.

    Sidik K, Jonkman JN. A simple confidence interval for meta-analysis. Stat Med. 2002;21(21):3153-9.

    Not in our library10.1002/sim.1262

  11. 11.

    Knapp G, Hartung J. Improved tests for a random effects meta-regression with a single covariate. Stat Med. 2003;22(17):2693-710.

    Not in our library10.1002/sim.1482

  12. 12.
    The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method · 2014
    No notice
  13. 13.

    Turner RM, Davey J, Clarke MJ, Thompson SG, Higgins JPT. Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane Database of Systematic Reviews. Int J Epidemiol. 2012;41(3):818-27.

    Not in our library10.1093/ije/dys041

  14. 14.
    A Re-Analysis of the Cochrane Library Data: The Dangers of Unobserved Heterogeneity in Meta-Analyses · 2013
    No notice
  15. 15.

    Higgins JPT, Thompson SG. Controlling the risk of spurious findings from meta-regression. Stat Med. 2004;23(11):1663-82.

    Not in our library10.1002/sim.1752

  16. 16.

    European Commission. Communication from the Commission on Regulation (EC) No 141/2000 of the European Parliament and of the Council of 16 December 1999 on orphan medicinal products. Off J Eur Union. 2003;46(C178):2-8.

    Not in our library

  17. 17.

    Gagne JJ, Thompson L, O’Keefe K, Kesselheim AS. Innovative research methods for studying treatments for rare diseases: methodological review. BMJ. 2014;349:6802.

    Not in our library10.1136/bmj.g6802

  18. 18.

    European Medicines Agency (EMEA). Guideline on clinical trials in small populations. CHMP/EWP/83561/2005. 2006. http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2009/09/WC500003615.pdf.

    Not in our library

  19. 19.

    Korn EL, McShane LM, Freidlin B. Statistical challenges in the evaluation of treatments for small patient populations. Sci Transl Med. 2013; 178. doi:10.1126/scitranslmed.3004018.

    Not in our library10.1126/scitranslmed.3004018

  20. 20.

    Kesselheim AS, Myers JA, Avorn J. Characteristics of clinical trials to support approval of orphan vs nonorphan drugs for cancer. J Am Med Assoc. 2011;305(22):2320-6.

    Not in our library10.1001/jama.2011.769

  21. 21.

    IntHout J, Ioannidis JPA, Borm GF, Goeman JJ. Small studies are more heterogeneous than large ones: a meta-meta-analysis. J Clin Epidemiol. 2015;68(8):860-9.

    Not in our library10.1016/j.jclinepi.2015.03.017

  22. 22.

    Prakken B, Albani S, Martini A. Juvenile idiopathic arthritis. The Lancet. 2011;377(9783):2138-149.

    Not in our library10.1016/s0140-6736(11)60244-4

  23. 23.

    Hinks A, Martin P, Flynn E, Eyre S, Packham J, Barton A, et al. Association of the CCR5 gene with juvenile idiopathic arthritis. Genes & Immunity. 2010;11(7):584-9.

    Not in our library10.1038/gene.2010.25

  24. 24.

    Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21(11):1539-58.

    Not in our library10.1002/sim.1186

  25. 25.

    DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7(3):177-88.

    Not in our library10.1016/0197-2456(86)90046-2

  26. 26.

    Böhning D, Malzahn U, Dietz E, Schlattmann P, Viwatwongkasem C, Biggeri A. Some general points in estimating heterogeneity variance with the DerSimonian-Laird estimator. Biostat. 2002;3(4):445-57.

    Not in our library10.1093/biostatistics/3.4.445

  27. 27.

    Raudenbush SW, Cooper H, Hedges LV, Valentine JC. Analyzing effect sizes: random-effects models. The handbook of research synthesis and meta-analysis. 2009.

    Not in our library

  28. 28.

    Paule RC, Mandel J. Consensus values and weighting factors. J Res Natl Bur Stand. 1982;87(5):377-85.

    Not in our library10.6028/jres.087.022

  29. 29.

    Rukhin AL, Biggerstaff BJ, Vangel MG. Restricted maximum-likelihood estimation of a common mean and the Mandel-Paule algorithm. J Stat Plan Infer. 2000;83(2):319-30.

    Not in our library10.1016/s0378-3758(99)00098-1

  30. 30.

    Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Soft.2010;36(3) doi:10.18637/jss.v036.i03.

    Not in our library

  31. 31.

    Schwarzer G. Meta: Meta-analysis with R. R package version 3.7-1. 2014. http://CRAN.R-project.org/package=meta.

    Not in our library

  32. 32.

    Hartung J. An alternative method for meta-analysis. Biom J. 1999;41(8):901-16.

    Not in our library10.1002/(sici)1521-4036(199912)41:8<901::aid-bimj901>3.0.co;2-w

  33. 33.

    Jackson D, Riley RD. A refined method for multivariate meta-analysis and meta-regression. Stat Med. 2014;33(4):541-54.

    Not in our library10.1002/sim.5957

  34. 34.

    Dawid AP. The well-calibrated Bayesian. J Am Stat Assoc. 1982;77(379):605-10.

    Not in our library10.1080/01621459.1982.10477856

  35. 35.

    Bradburn MJ, Deeks JJ, Berlin JA, Localio AR. Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events. Stat Med. 2007;26(1):53-77.

    Not in our library10.1002/sim.2528

  36. 36.

    Kuß O. Statistical methods for meta-analyses including information from studies without any events—add nothing to nothing and succeed nevertheless. Stat Med. 2015;34(7):1097-116.

    Not in our library10.1002/sim.6383

  37. 37.

    R Core Team. R: A Language and Environment for Statistical Computing. 2014.

    Not in our library

  38. 38.

    Sánchez-Meca J, Marín-Martínez F. Confidence intervals for the overall effect size in random-effects meta-analysis. Psychol Methods. 2008;13(1):31-48.

    Not in our library10.1037/1082-989x.13.1.31

  39. 39.

    Sidik K, Jonkman JN. Authors’ reply. Stat Med. 2004;23(1):159-62.

    Not in our library10.1002/sim.1729

  40. 40.

    Viechtbauer W. Confidence intervals for the amount of heterogeneity in meta-analysis. Stat Med. 2007;26(1):37-52.

    Not in our library10.1002/sim.2514

  41. 41.

    Gonnermann A, Framke T, Großhennig A, Koch A. No solution yet for combining two independent studies in the presence of heterogeneity. Stat Med. 2015;34(16):2476-80.

    Not in our library10.1002/sim.6473

  42. 42.

    Sutton AJ, Abrams KR. Bayesian methods in meta-analysis and evidence synthesis. Stat Methods Med Res. 2001;10(4):277-303.

    Not in our library10.1191/096228001678227794

  43. 43.
    Predictive distributions for between‐study heterogeneity and simple methods for their application in Bayesian meta‐analysis · 2014
    No notice
  44. 44.

    Innovative methodology for small populations research (InSPiRe). http://www.warwick.ac.uk/inspire.

    Not in our library

Cited by papers we hold 4

Reads

Read by people on this page

None yet.

Checked or read by AI through naulon

None yet.

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.

Are you an author?

Is this your paper?

Claim it to add your note and be paid when agents cite it. Free.