Multivariate meta‐analysis: Potential and promise

Abstract

The multivariate random effects model is a generalization of the standard univariate model. Multivariate meta‐analysis is becoming more commonly used and the techniques and related computer software, although continually under development, are now in place. In order to raise awareness of the multivariate methods, and discuss their advantages and disadvantages, we organized a one day ‘Multivariate meta‐analysis’ event at the Royal Statistical Society. In addition to disseminating the most recent developments, we also received an abundance of comments, concerns, insights, critiques and encouragement. This article provides a balanced account of the day's discourse. By giving others the opportunity to respond to our assessment, we hope to ensure that the various view points and opinions are aired before multivariate meta‐analysis simply becomes another widely used de facto method without any proper consideration of it by the medical statistics community. We describe the areas of application that multivariate meta‐analysis has found, the methods available, the difficulties typically encountered and the arguments for and against the multivariate methods, using four representative but contrasting examples. We conclude that the multivariate methods can be useful, and in particular can provide estimates with better statistical properties, but also that these benefits come at the price of making more assumptions which do not result in better inference in every case. Although there is evidence that multivariate meta‐analysis has considerable potential, it must be even more carefully applied than its univariate counterpart in practice. Copyright © 2011 John Wiley & Sons, Ltd.

Shown for reading. Its publisher allows quoting it, not republishing it.

References 75

1. 1.

   Egger M, Smith GD. Misleading meta-analysis. British Medical Journal. 1995;310:752-754.

   Not in our library10.1136/bmj.310.6982.752

2. 2.

   Egger M, Smith GD. Meta-analysis: potentials and promise. British Medical Journal. 1997;315:1371-1375.

   Not in our library10.1136/bmj.315.7119.1371

3. 3.

   Eysenck HJ. Meta-analysis and its problems. British Medical Journal. 1994;309:789-792.

   Not in our library10.1136/bmj.309.6957.789

4. 4.

   Normand ST. Meta-analysis: formulating, evaluating, combining and reporting. Statistics in Medicine. 1999;18:321-359.

   Not in our library10.1002/(sici)1097-0258(19990215)18:3<321::aid-sim28>3.0.co;2-p

5. 5.

   Reitsma JB, Glas AS, Rutjes AW, Scholten RJ, Bossuyt PM, Zwinderman AH. Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. Journal of Clinical Epidemiology. 2005;58:982-990.

   Not in our library10.1016/j.jclinepi.2005.02.022

6. 6.

   Higgins JPT, Whitehead A, Turner RM, Omar RZ, Thompson SG. Meta-analysis of continuous outcome data from individual patients. Statistics in Medicine. 2001;20:2219-2241.

   Not in our library10.1002/sim.918

7. 7.

   Turner RM, Omar RZ, Yang M, Goldstein H, Thompson SG. A multilevel model framework for meta-analysis of clinical trials with binary outcomes. Statistics in Medicine. 2000;19:3417-3432.

   Not in our library10.1002/1097-0258(20001230)19:24<3417::aid-sim614>3.0.co;2-l

8. 8.

   Whitehead A, Omar RZ, Higgins JPT, Savaluny E, Turner RM, Thompson SG. Meta-analysis of ordinal outcomes using individual patient data. Statistics in Medicine. 2001;20:2241-2243.

   Not in our library10.1002/sim.919

9. 9.

   Tudur Smith C, Williamson PR, Marson AG. Investigating heterogeneity in an individual patient data meta-analysis of time to event outcomes. Statistics in Medicine. 2005;24:1307-1319.

   Not in our library10.1002/sim.2050

10. 10.

    Systematically missing confounders in individual participant data meta-analysis of observational studies. Statistics in Medicine. 2009;28:1218-1237.

    Not in our library10.1002/sim.3540

11. 11.

    van Houwelingen HC, Arends LR, Stijnen T. Advanced methods in meta-analysis: multivariate approach and meta-regression. Statistics in Medicine. 2002;21:589-624.

    Not in our library10.1002/sim.1040

12. 12.

    Ishak KJ, Platt RW, Joseph L, Hanley JA. Impact of approximating or ignoring within-study covariances in multivariate meta-analyses. Statistics in Medicine. 2008;27:670-686.

    Not in our library10.1002/sim.2913

13. 13.

    Riley R. Multivariate meta-analysis: the effect of ignoring within-study correlation. Journal of the Royal Statistical Society, Series A. 2009;172:789-811.

    Not in our library

14. 14.

    White IR. Multivariate meta-analysis. The Stata Journal. 2009;9:40-56.

    Not in our library

15. 15.

    Gleser LJ, Olkin I, Cooper H, Hedges LV. Stochastically dependent effect sizes. The Handbook of Research Synthesis. 1994.

    Not in our library

16. 16.

    Raudenbush SW, Becker BJ, Kalaian H. Modeling multivariate effect sizes. Psychological Bulletin. 1988;1039:111-120.

    Not in our library

17. 17.

    Kalaian HA, Raudenbush SW. A multivariate mixed linear model for meta-analysis. Psychological Methods. 1996;1:227-235.

    Not in our library

18. 18.

    Hedges LV, Olkin I. Statistical Methods for Meta-analysis. 1994.

    Not in our library

19. 19.

    Berkey CS, Antczak-Bouckoms A, Hoaglin DC, Mosteller F, Pihlstrom BL. Multiple outcomes meta-analysis of treatments for periodontal disease. Journal of Dental Research. 1995;74:1030-1039.

    Not in our library10.1177/00220345950740040201

20. 20.

    Berkey CS, Hoaglin DC, Antczak-Bouckoms A, Mosteller F, Colditz GA. Meta-analysis of multiple outcomes by regression with random effects. Statistics in Medicine. 1998;17:2537-2550.

    Not in our library10.1002/(sici)1097-0258(19981130)17:22<2537::aid-sim953>3.0.co;2-c

21. 21.

    Sohn SY. Multivariate meta-analysis with potentially correlated marketing study results. Naval Research Logistics. 2000;47:500-510.

    Not in our library

22. 22.

    Daniels MJ, Hughes MD. Meta-analysis for the evaluation of potential surrogate markers. Statistics in Medicine. 1997;16:1965-1982.

    Not in our library10.1002/(sici)1097-0258(19970915)16:17<1965::aid-sim630>3.0.co;2-m

23. 23.

    Gail MH, Pfeiffer R, van Houwelingen HC, Carroll RJ. On meta-analytic assessment of surrogate outcomes. Biostatistics. 2000;1:231-246.

    Not in our library10.1093/biostatistics/1.3.231

24. 24.

    Thompson JR, Minelli C, Abrams KR, Tobin MD, Riley RD. Meta-analysis of genetic studies using Mendelian randomization—a multivariate approach. Statistics in Medicine. 2005;24:2241-2254.

    Not in our library10.1002/sim.2100

25. 25.

    Riley RD, Abrams KR, Lambert PC, Sutton AJ, Thompson JR. An evaluation of bivariate random effects meta-analysis for the joint synthesis of two correlated outcomes. Statistics in Medicine. 2007;26:78-97.

    Not in our library10.1002/sim.2524

26. 26.

    Hamza TH, Arends LR, van Houwelingen HC, Stijnen T. Multivariate random effects meta-analysis of diagnostic tests with multiple thresholds. BMC Medical Research Methodology. 2009;9:73.

    Not in our library10.1186/1471-2288-9-73

27. 27.

    Jones AP, Riley RD, Williamson PR, Whitehead A. Meta-analysis of longitudinal data. Clincal Trials. 2009;6:16-27.

    Not in our library10.1177/1740774508100984

28. 28.

    Chu H, Cole SR. Bivariate meta-analysis of sensitivity and specificity with sparse data: a generalized linear mixed model approach. Journal of Clinical Epidemiology. 2006;59:1331-1333.

    Not in our library10.1016/j.jclinepi.2006.06.011

29. 29.

    Hamza TH, van Houwelingen HC, Stijnen T. The binomial distribution of meta-analysis was preferred to model within-study variability. Journal of Clinical Epidemiology. 2008;61:41-51.

    Not in our library10.1016/j.jclinepi.2007.03.016

30. 30.

    Harbord RM, Whiting P. Metandi: meta-analysis of diagnostic accuracy using hierarchical logistic regression. The Stata Journal. 2009;9:211-229.

    Not in our library

31. 31.

    Takwoingi Y. 2008.

    Not in our library

32. 32.

    Harbord RM, Deeks JJ, Egger M, Whiting P, Sterne JAC. A unification of models for meta-analysis of diagnostic accuracy studies. Biostatistics. 2007;8:239-251.

    Not in our library10.1093/biostatistics/kxl004

33. 33.

    Rutter CM, Gatsonis CA. A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations. Statistics in Medicine. 2001;20:2865-2884.

    Not in our library10.1002/sim.942

34. 34.

    Putter H, Fiocco M, Stijnen T. Meta-analysis of diagnostic test accuracy studies with multiple thresholds using survival methods. Biometrical Journal. 2010;52:95-110.

    Not in our library10.1002/bimj.200900073

35. 35.

    Chu H, Nie L, Cole SR, Poole C. Meta-analysis of diagnostic accuracy studies accounting for disease prevalence: alternative parameterizations and model selection. Statistics in Medicine. 2009;28:2384-2399.

    Not in our library10.1002/sim.3627

36. 36.

    Paul M, Riebler A, Bachmann LM, Rue H, Held L. Bayesian bivariate meta-analysis of diagnostic test studies using integrated nested Laplace approximations. Statistics in Medicine. 2010;29:1325-1339.

    Not in our library10.1002/sim.3858

37. 37.

    Kertai MD, Boersma E, Bax JJ, Heijenbrok-Kal MH, Hunink MGM, L'talien GJ, Roelandt JRTC, van Urk H, Poldermans D. A meta-analysis comparing the prognostic accuracy of six diagnostic tests for predicting perioperative cardiac risk in patients undergoing major vascular surgery. Heart. 2003;89:1327-1334.

    Not in our library10.1136/heart.89.11.1327

38. 38.

    Berkey CS, Anderson JJ, Hoaglin DC. Multiple-outcome meta-analysis of clinical trials. Statistics in Medicine. 1996;15:537-557.

    Not in our library10.1002/(sici)1097-0258(19960315)15:5<537::aid-sim176>3.0.co;2-s

39. 39.

    Arends LR, Hunink MG, Stijnen T. Meta-analysis of summary survival curve data. Statistics in Medicine. 2008;27:4381-4396.

    Not in our library10.1002/sim.3311

40. 40.

    van Houwelingen HC, Zwinderman KH, Stijnen T. A bivariate approach to meta-analysis. Statistics in Medicine. 1993;12:2273-2284.

    Not in our library10.1002/sim.4780122405

41. 41.

    Komuro H, Valentine MB, Rowe ST, Kidd VJ, Makino S, Brodeur GM, Cohn SL, Look AT. Fluorescence in situ hybridization analysis of chromosome 1p36 deletions in human MYCN amplified neuroblastoma. Journal of Pediatric Surgery. 1998;33:1695-1698.

    Not in our library10.1016/s0022-3468(98)90612-1

42. 42.

    Tandon S, Tudur-Smith C, Riley RD, Boyd MT, Jones TM. A systematic review of p53 as a prognostic factor of survival in squamous cell carcinoma of the four main anatomical subsites of the head and neck. Cancer Epidemiology, Biomarkers and Prevention. 2010;19:574-587.

    Not in our library10.1158/1055-9965.epi-09-0981

43. 43.

    The Emerging Risk Factors Collaboration. Diabetes mellitus, fasting blood glucose concentration, and risk of vascular disease: a collaborative meta-analysis of 102 prospective studies. Lancet. 2010;375:2215-2222.

    Not in our library10.1016/s0140-6736(10)60484-9

44. 44.

    Salanti G, Ades AE, Higgins JPT, Ioannidis JPA. Evaluation of networks of randomised trials. Statistical Methods in Medical Research. 2008;17:3105-3124.

    Not in our library10.1177/0962280207080643

45. 45.

    Lu G, Ades A. Assessing evidence consistency in mixed treatment comparisons. Journal of the American Statistical Association. 2006;101:447-459.

    Not in our library

46. 46.

    Lumley T. Network meta-analysis for indirect treatment comparisons. Statistics in Medicine. 2008;21:12313-12324.

    Not in our library10.1002/sim.1201

47. 47.

    Ritz J, Demidenko E, Spiegelman D. Multivariate meta-analysis for data consortia, individual patient meta-analysis, and pooling projects. Journal of Statistical Planning and Inference. 2008;138:1919-1933.

    Not in our library

48. 48.

    Baker R, Jackson D. A new approach to outliers in meta-analysis. Health Care in Management Science. 2008;11:121-131.

    Not in our library10.1007/s10729-007-9041-8

49. 49.

    Lee KJ, Thompson SG. Flexible parametric models for random effects distributions. Statistics in Medicine. 2008;27:418-434.

    Not in our library10.1002/sim.2897

50. 50.

    Jackson D. The significance level of the standard test for a treatment effect in meta-analysis. Statistics in Biopharmaceutical Research. 2009;1:92-100.

    Not in our library

51. 51.
    Bivariate random-effects meta-analysis and the estimation of between-study correlation · 2007

    No notice
52. 52.

    Jackson D, White IR, Thompson SG. Extending DerSimonian and Laird's methodology to perform multivariate random effects meta-analyses. Statistics in Medicine. 2010;29:1282-1297.

    Not in our library10.1002/sim.3602

53. 53.

    Jennrich RI, Schluchter MD. Unbalanced repeated-measures models with structured covariance matrices. Biometrics. 1986;42:805-820.

    Not in our library

54. 54.

    Dersimonian R, Laird N. Meta-analysis in clinical trials. Controlled Clinical Trials. 1986;7:177-188.

    Not in our library10.1016/0197-2456(86)90046-2

55. 55.

    Kenward MG, Roger JH. Small sample inference for fixed effects from restricted maximum likelihood. Biometrics. 1997;53:983-997.

    Not in our library

56. 56.

    Hardy RJ, Thompson SG. A likelihood approach to meta-analysis with random effects. Statistics in Medicine. 1996;15:619-629.

    Not in our library10.1002/(sici)1097-0258(19960330)15:6<619::aid-sim188>3.0.co;2-a

57. 57.

    Jackson D, Bowden J, Baker R. Hoes does the DerSimonian and Laird procedure for random effects meta-analysis compare with its more efficient but harder to compute counterparts? Journal of Statistical Planning and Inference. 2010;140:961-970.

    Not in our library

58. 58.

    Lunn DJ, Thomas A, Best N, Spiegelhalter D. WinBUGS—a Bayesian modelling framework: concepts, structure, and extensibility. Statistics in Computing. 2000;19:325-337.

    Not in our library

59. 59.

    Nam IS, Mengerson K, Garthwaite P. Multivariate meta-analysis. Statistics in Medicine. 2003;22:2309-2333.

    Not in our library10.1002/sim.1410

60. 60.

    Lambert PC, Sutton AJ, Burton PR, Abrams KR, Jones DR. How vague is vague? A simulation study of the impact of the use of vague prior distributions in MCMC using WinBUGS. Statistics in Medicine. 2010;24:2401-2428.

    Not in our library10.1002/sim.2112

61. 61.

    Cox DR. Principles of Statistical Inference. 2006.

    Not in our library

62. 62.

    Riley RD, Dodd SR, Craig JV, Thompson JR, Williamson PR. Meta-analysis of diagnostic test studies using individual patient data and aggregate data. Statistics in Medicine. 2008;27:6111-6136.

    Not in our library10.1002/sim.3441

63. 63.
    A Re-Evaluation of Random-Effects Meta-Analysis · 2008

    No notice
64. 64.

    Hand D. Evaluating diagnostic tests: the area under the ROC curve and the balance of errors. Statistics in Medicine. 2010;29:1502-1510.

    Not in our library10.1002/sim.3859

65. 65.

    Simel DL, Bossuyt PMM. Differences between univariate and bivariate models for summarizing diagnostic accuracy may not be large. Journal of Clinical Epidemiology. 2009;62:1292-1300.

    Not in our library10.1016/j.jclinepi.2009.02.007

66. 66.

    Riley RD, Sutton AJ, Abrams KR, Lambert PC. Sensitivity analyses allowed more appropriate and reliable meta-analysis conclusions for multiple outcomes when missing data was present. Journal of Clinical Epidemiology. 2004;57:911-924.

    Not in our library10.1016/j.jclinepi.2004.01.018

67. 67.

    Jackson D, Copas J, Sutton AJ. Modelling reporting bias: the operative mortality rate for ruptured abdominal aortic aneurysm repair. Journal of the Royal Statistical Society, Series A. 2005;168:737-752.

    Not in our library

68. 68.

    Olkin I, Trikalinos TA. A method for the meta-analysis of mutually exclusive binary outcomes. Statistics in Medicine. 2008;27:4279-4300.

    Not in our library10.1002/sim.3299

69. 69.

    Raudenbush SW, Becker BJ, Kalaian H. Modeling multivariate effect sizes. Psychological Bulletin. 1988;103:111-120.

    Not in our library

70. 70.

    Berrington A, Cox DR. Generalized least squares for the synthesis of correlated information. Biostatistics. 2003;4:423-431.

    Not in our library10.1093/biostatistics/4.3.423

71. 71.

    Riley RD, Thompson JR, Abrams KR. An alternative model for bivariate random-effects meta-analysis when the within-study correlations are unknown. Biostatistics. 2008;9:172-186.

    Not in our library10.1093/biostatistics/kxm023

72. 72.

    Hedges LV, Tipton E, Johnson MC. Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods. 2010;1:39-65.

    Not in our library10.1002/jrsm.5

73. 73.

    Thompson SG, Sharp SJ. Explaining heterogeneity in meta-analysis: a comparison of methods. Statistics in Medicine. 1999;18:2693-2708.

    Not in our library10.1002/(sici)1097-0258(19991030)18:20<2693::aid-sim235>3.0.co;2-v

74. 74.

    Jackson D. The significance level of meta-regression's standard hypothesis test. Communications in Statistics—Theory and Methods. 2008;37:1576-1590.

    Not in our library

75. 75.

    Higgins JPT, Thompson SG. Quantifying heterogeneity in meta-analysis. Statistics in Medicine. 2002;21:1539-1558.

    Not in our library10.1002/sim.1186

Cited by papers we hold 18

• Caffeine, Coffee, Tea and Risk of Rheumatoid Arthritis: Systematic Review and Dose-Response Meta-analysis of Prospective Cohort Studies · 2022
  Not checked
• Mindfulness-based programmes for mental health promotion in adults in nonclinical settings: A systematic review and meta-analysis of randomised controlled trials · 2021
  No notice
• A multi-cohort consortium for GEnder-Sensitive Analyses of mental health trajectories and implications for prevention (GESA) in the general population in Germany · 2020
  Not checked
• Individual participant data meta‐analysis to examine interactions between treatment effect and participant‐level covariates: Statistical recommendations for conduct and planning · 2020
  No notice
• What exercise prescription is optimal to improve body composition and cardiorespiratory fitness in adults living with obesity? A network meta‐analysis · 2020
  No notice
• A Guide to Conducting a Meta-Analysis with Non-Independent Effect Sizes · 2019
  No notice
• When should meta‐analysis avoid making hidden normality assumptions? · 2018
  No notice
• Multivariate and network meta-analysis of multiple outcomes and multiple treatments: rationale, concepts, and examples · 2017
  No notice
• Meta-evaluation of meta-analysis: ten appraisal questions for biologists · 2017
  No notice
• Meta-analysis using individual participant data: one-stage and two-stage approaches, and why they may differ · 2016
  No notice
• Accuracy of non-invasive prenatal testing using cell-free DNA for detection of Down, Edwards and Patau syndromes: a systematic review and meta-analysis · 2016
  No notice
• A graphical tool for locating inconsistency in network meta-analyses · 2013
  No notice
• Individual Participant Data Meta-Analysis for a Binary Outcome: One-Stage or Two-Stage? · 2013
  No notice
• Graphical Tools for Network Meta-Analysis in STATA · 2013
  No notice
• Reducing and meta-analysing estimates from distributed lag non-linear models · 2013
  No notice
• Quantifying the impact of between‐study heterogeneity in multivariate meta‐analyses · 2012
  No notice
• Multivariate meta‐analysis for non‐linear and other multi‐parameter associations · 2012
  No notice
• Consistency and inconsistency in network meta‐analysis: model estimation using multivariate meta‐regression · 2012
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.