Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity‐score matched samples

Abstract

The propensity score is a subject's probability of treatment, conditional on observed baseline covariates. Conditional on the true propensity score, treated and untreated subjects have similar distributions of observed baseline covariates. Propensity‐score matching is a popular method of using the propensity score in the medical literature. Using this approach, matched sets of treated and untreated subjects with similar values of the propensity score are formed. Inferences about treatment effect made using propensity‐score matching are valid only if, in the matched sample, treated and untreated subjects have similar distributions of measured baseline covariates. In this paper we discuss the following methods for assessing whether the propensity score model has been correctly specified: comparing means and prevalences of baseline characteristics using standardized differences; ratios comparing the variance of continuous covariates between treated and untreated subjects; comparison of higher order moments and interactions; five‐number summaries; and graphical methods such as quantile–quantile plots, side‐by‐side boxplots, and non‐parametric density plots for comparing the distribution of baseline covariates between treatment groups. We describe methods to determine the sampling distribution of the standardized difference when the true standardized difference is equal to zero, thereby allowing one to determine the range of standardized differences that are plausible with the propensity score model having been correctly specified. We highlight the limitations of some previously used methods for assessing the adequacy of the specification of the propensity‐score model. In particular, methods based on comparing the distribution of the estimated propensity score between treated and untreated subjects are uninformative. Copyright © 2009 John Wiley & Sons, Ltd.

Shown for reading. Its publisher allows quoting it, not republishing it.

References 47

1. 1.

   Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70:41-55.

   Not in our library

2. 2.

   Rosenbaum PR, Rubin DB. Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association. 1984;79:516-524.

   Not in our library

3. 3.

   Austin PC, Mamdani MM. A comparison of propensity score methods: a case-study estimating the effectiveness of post-AMI statin use. Statistics in Medicine. 2006;25:2084-2106.

   Not in our library10.1002/sim.2328

4. 4.

   Weitzen S, Lapane KL, Toledano AY, Hume AL, Mor V. Principles for modeling propensity scores in medical research: a systematic literature review. Pharmacoepidemiology and Drug Safety. 2004;13:841-853.

   Not in our library10.1002/pds.969

5. 5.

   Shah BR, Laupacis A, Hux JE, Austin PC. Propensity score methods give similar results to traditional regression modeling in observational studies: a systematic review. Journal of Clinical Epidemiology. 2005;58:550-559.

   Not in our library10.1016/j.jclinepi.2004.10.016

6. 6.

   Stürmer T, Joshi M, Glynn RJ, Avorn J, Rothman KJ, Schneeweiss S. A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. Journal of Clinical Epidemiology. 2006;59:437-447.

   Not in our library10.1016/j.jclinepi.2005.07.004

7. 7.

   Austin PC. A critical appraisal of propensity score matching in the medical literature from 1996 to 2003. Statistics in Medicine. 2008;27:2037-2049.

   Not in our library10.1002/sim.3150

8. 8.

   Austin PC. Propensity-score matching in the cardiovascular surgery literature from 2004 to 2006: a systematic review and suggestions for improvement. Journal of Thoracic and Cardiovascular Surgery. 2007;134:1128-1135.

   Not in our library10.1016/j.jtcvs.2007.07.021

9. 9.

   Austin PC. A report card on propensity-score matching in the cardiology literature from 2004 to 2006: results of a systematic review. Circulation: Cardiovascular Quality and Outcomes. 2008;1:62-67.

   Not in our library10.1161/circoutcomes.108.790634

10. 10.

    Rubin DB. Using propensity scores to help design observational studies: application to the tobacco litigation. Health Services and Outcomes Research Methodology. 2001;2:169-188.

    Not in our library

11. 11.

    Rubin DB. On principles for modeling propensity scores in medical research. Pharmacoepidemiology and Drug Safety. 2004;13:855-857.

    Not in our library10.1002/pds.968

12. 12.

    Ho DE, Imai K, King G, Stuart EA. Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Analysis. 2007;15:199-236.

    Not in our library

13. 13.

    Austin PC, Mamdani MM, Juurlink DN, Alter DA, Tu JV. Missed opportunities in the secondary prevention of myocardial infarction: an assessment of the effects of statin underprescribing on mortality. American Heart Journal. 2006;151:969-975.

    Not in our library10.1016/j.ahj.2005.06.034

14. 14.

    Austin PC, Tu JV. Comparing clinical data with administrative data for producing AMI report cards. Journal of the Royal Statistical Society—Series A (Statistics in Society). 2006;169:115-126.

    Not in our library

15. 15.

    Austin PC. A comparison of classification and regression trees, logistic regression, generalized additive models, and multivariate adaptive regression splines for predicting AMI mortality. Statistics in Medicine. 2007;26:2937-2957.

    Not in our library10.1002/sim.2770

16. 16.

    Tu JV, Donovan LR, Lee DS, Austin PC, Ko DT, Wang JT, Newman AM. Quality of Cardiac Care in Ontario. 2004.

    Not in our library

17. 17.

    Austin PC, Grootendorst P, Anderson GM. A comparison of the ability of different propensity score models to balance measured variables between treated and untreated subjects: a Monte Carlo study. Statistics in Medicine. 2007;26:734-753.

    Not in our library10.1002/sim.2580

18. 18.

    Austin PC. Some methods of propensity-score matching had superior performance to others: results of an empirical investigation and Monte Carlo simulations. Biometrical Journal. 2009;51:171-184.

    Not in our library10.1002/bimj.200810488

19. 19.

    Moher D, Schulz KF, Altman D, for the CONSORT Group. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. Journal of the American Medical Association. 2001;285:1787-1991.

    Not in our library10.7547/87507315-91-8-437

20. 20.

    Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, Gotzsche PC, Lang T, for the CONSORT Group. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Annals of Internal Medicine. 2001;134:663-694.

    Not in our library10.7326/0003-4819-134-8-200104170-00012

21. 21.

    Flury BK, Riedwyl H. Standard distance in univariate and multivariate analysis. The American Statistician. 1986;40:249-251.

    Not in our library

22. 22.

    Rosenbaum PR, Rubin DB. Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. The American Statistician. 1985;39:33-38.

    Not in our library

23. 23.

    Normand SLT, Landrum MB, Guadagnoli E, Ayanian JZ, Ryan TJ, Cleary PD, McNeil BJ. Validating recommendations for coronary angiography following an acute myocardial infarction in the elderly: a matched analysis using propensity scores. Journal of Clinical Epidemiology. 2001;54:387-398.

    Not in our library10.1016/s0895-4356(00)00321-8

24. 24.

    Ahmed A, Perry GJ, Fleg JL, Love TE, Goff DC, Kitzman DW. Outcomes in ambulatory chronic systolic and diastolic heart failure: a propensity score analysis. American Heart Journal. 2006;152:956-966.

    Not in our library10.1016/j.ahj.2006.06.020

25. 25.

    Ahmed A, Husain A, Love TE, Gambassi G, Dell'Italia LJ, Francis GS, Gheorghiade M, Allman RM, Meleth S, Bourge RC. Heart failure, chronic diuretic use, and increase in mortality and hospitalization: an observational study using propensity score methods. European Heart Journal. 2006;27:1431-1439.

    Not in our library10.1093/eurheartj/ehi890

26. 26.

    Cohen J. Statistical Power Analysis for the Behavioral Sciences. 1988.

    Not in our library

27. 27.

    Hedges LV, Olkin I. Statistical Methods for Meta-Analysis. 1985.

    Not in our library

28. 28.

    Austin PC. Type I error rates, coverage of confidence intervals, and variance estimation in propensity-score matched analyses. The International Journal of Biostatistics. 2009;5(1).

    Not in our library10.2202/1557-4679.1146

29. 29.

    Imai K, King G, Stuart EA. Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society, Series A (Statistics in Society). 2008;171:481-502.

    Not in our library

30. 30.

    Rosner B. Fundamentals of Biostatistics. 1995.

    Not in our library

31. 31.

    Harrell FE. Regression Modeling Strategies. 2001.

    Not in our library

32. 32.

    Hoaglin DC, Mosteller F, Tukey JW. Understanding Robust and Exploratory Data Analysis. 1983.

    Not in our library

33. 33.

    Casella G, Berger RL. Statistical Inference. 1990.

    Not in our library

34. 34.

    Weitzen S, Lapane KL, Toledano AY, Hume AL, Mor V. Weaknesses of goodness-of-fit tests for evaluating propensity score models: the case of the omitted confounder. Pharmacoepidemiology and Drug Safety. 2005;14:227-238.

    Not in our library10.1002/pds.986

35. 35.

    Senn S. Testing for baseline balance in clinical trials. Statistics in Medicine. 1994;13:1715-1726.

    Not in our library10.1002/sim.4780131703

36. 36.

    Senn SJ. Covariate imbalance and random allocation in clinical trials. Statistics in Medicine. 1989;8:467-475.

    Not in our library10.1002/sim.4780080410

37. 37.

    Altman DG, Dore CJ. Baseline comparisons in randomized clinical trials. Statistics in Medicine. 1991;10:797-802.

    Not in our library10.1002/sim.4780100514

38. 38.

    Lavori PW, Louis TA, Bailar JC, Polansky M. Designs for experiments—parallel comparisons of treatment. New England Journal of Medicine. 1983;309:1291-1298.

    Not in our library10.1056/nejm198311243092105

39. 39.

    Gail MH, Wieand S, Piantadosi S. Biased estimates of treatment effect in randomized experiments with nonlinear regressions and omitted covariates. Biometrika. 1984;7:431-444.

    Not in our library

40. 40.

    Austin PC, Zwarenstein M, Manca A, Juurlink DN, Stanbrook MB. Handling of baseline covariates in randomized controlled trials: a review of trials published in leading medical journals. Journal of Clinical Epidemiology.

    Not in our library10.1016/j.jclinepi.2009.06.002

Cited by papers we hold 25

• Matching-adjusted indirect comparison between garadacimab and lanadelumab for the long-term prophylactic treatment of patients with hereditary angioedema · 2025
  Not checked
• Postacute sequelae of COVID-19 at 2 years · 2023
  No notice
• Survival outcomes of combined hepatocellular‐cholangiocarcinoma compared with intrahepatic cholangiocarcinoma: A SEER population‐based cohort study · 2022
  Not checked
• Lowering serum homocysteine in H-type hypertensive patients with atrial fibrillation after radiofrequency catheter ablation to prevent atrial fibrillation recurrence · 2022
  Not checked
• Impact of maternal neuraxial labor analgesia exposure on offspring's neurodevelopment: A longitudinal prospective cohort study with propensity score matching · 2022
  Not checked
• The influence of mode of anaesthesia on perioperative outcomes in people with hip fracture: a prospective cohort study from the National Hip Fracture Database for England, Wales and Northern Ireland · 2022
  Not checked
• Trajectory of long covid symptoms after covid-19 vaccination: community based cohort study · 2022
  No notice
• Post-covid syndrome in individuals admitted to hospital with covid-19: retrospective cohort study · 2021
  No notice
• Major adverse cardiovascular and limb events in patients with diabetes and concomitant peripheral artery disease treated with sodium glucose cotransporter 2 inhibitor versus dipeptidyl peptidase-4 inhibitor · 2020
  Not checked
• Hemoadsorption of cytokines by CytoSorb filter: a simulation study without human factor—pilot is the difference · 2020
  Not checked
• Comparative host-coronavirus protein interaction networks reveal pan-viral disease mechanisms · 2020
  No notice
• Propensity Score Methods in Health Technology Assessment: Principles, Extended Applications, and Recent Advances · 2019
  No notice
• Evaluation of treatment response in adults with relapsing MOG-Ab-associated disease · 2019
  No notice
• International Comparison of Patient Characteristics and Quality of Care for Ischemic Stroke: Analysis of the China National Stroke Registry and the American Heart Association Get With The Guidelines––Stroke Program · 2018
  Not checked
• The reporting of studies conducted using observational routinely collected health data statement for pharmacoepidemiology (RECORD-PE) · 2018
  No notice
• Capnography sensor use is associated with reduction of adverse outcomes during gastrointestinal endoscopic procedures with sedation administration · 2017
  Not checked
• Double-adjustment in propensity score matching analysis: choosing a threshold for considering residual imbalance · 2017
  No notice
• Marital Status and Survival in Patients with Carcinoid Tumors · 2016
  Not checked
• Costs and Benefits Associated With Transradial Versus Transfemoral Percutaneous Coronary Intervention in China · 2016
  Not checked
• Bariatric Surgery in the United Kingdom: A Cohort Study of Weight Loss and Clinical Outcomes in Routine Clinical Care · 2015
  No notice

and 5 more

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.