The performance of different propensity-score methods for estimating differences in proportions (risk differences or absolute risk reductions) in observational studies

Abstract

Propensity score methods are increasingly being used to estimate the effects of treatments on health outcomes using observational data. There are four methods for using the propensity score to estimate treatment effects: covariate adjustment using the propensity score, stratification on the propensity score, propensity-score matching, and inverse probability of treatment weighting (IPTW) using the propensity score. When outcomes are binary, the effect of treatment on the outcome can be described using odds ratios, relative risks, risk differences, or the number needed to treat. Several clinical commentators suggested that risk differences and numbers needed to treat are more meaningful for clinical decision making than are odds ratios or relative risks. However, there is a paucity of information about the relative performance of the different propensity-score methods for estimating risk differences. We conducted a series of Monte Carlo simulations to examine this issue. We examined bias, variance estimation, coverage of confidence intervals, mean-squared error (MSE), and type I error rates. A doubly robust version of IPTW had superior performance compared with the other propensity-score methods. It resulted in unbiased estimation of risk differences, treatment effects with the lowest standard errors, confidence intervals with the correct coverage rates, and correct type I error rates. Stratification, matching on the propensity score, and covariate adjustment using the propensity score resulted in minor to modest bias in estimating risk differences. Estimators based on IPTW had lower MSE compared with other propensity-score methods. Differences between IPTW and propensity-score matching may reflect that these two methods estimate the average treatment effect and the average treatment effect for the treated, respectively.

Shown for reading. Its publisher allows quoting it, not republishing it.

References 34

1. 1.

   Schechtman E. Odds ratio, relative risk, absolute risk reduction, and the number needed to treat—which of these should we use? Value in Health. 2002;5:431-436.

   Not in our library10.1046/j.1524-4733.2002.55150.x

2. 2.

   Cook RJ, Sackett DL. The number needed to treat: a clinically useful measure of treatment effect. British Medical Journal. 1995;310:452-454.

   Not in our library10.1136/bmj.310.6977.452

3. 3.

   Jaeschke R, Guyatt G, Shannon H, Walter S, Cook D, Heddle N. Basis statistics for clinicians 3: assessing the effects of treatment: measures of association. Canadian Medical Association Journal. 1995;152:351-357.

   Not in our library

4. 4.

   Sinclair JC, Bracken MB. Clinically useful measures of effect in binary analyses of randomized trials. Journal of Clinical Epidemiology. 1994;47:881-889.

   Not in our library10.1016/0895-4356(94)90191-0

5. 5.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

6. 6.

   Austin PC, Grootendorst P, Normand SLT, Anderson GM. Conditioning on the propensity score can result in biased estimation of common measures of treatment effect: a Monte Carlo study. Statistics in Medicine. 2007;26:754-768.

   Not in our library10.1002/sim.2618

7. 7.

   Austin PC. The performance of different propensity score methods for estimating marginal odds ratios. Statistics in Medicine. 2007;26:3078-3094.

   Not in our library10.1002/sim.2781

8. 8.

   Austin PC. The performance of different propensity score methods for estimating relative risks. Journal of Clinical Epidemiology. 2008;61:537-545.

   Not in our library10.1016/j.jclinepi.2007.07.011

9. 9.

   Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70:41-55.

   Not in our library

10. 10.

    Rosenbaum PR, Rubin DB. Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association. 1984;79:516-524.

    Not in our library

11. 11.

    Rosenbaum PR. Model-based direct adjustment. The Journal of the American Statistician. 1987;82:387-394.

    Not in our library

12. 12.

    Austin PC. A critical appraisal of propensity score matching in the medical literature from 1996 to 2003. Statistics in Medicine. 2008;27:2037-2049.

    Not in our library10.1002/sim.3150

13. 13.

    Austin PC. Propensity-score matching in the cardiovascular surgery literature from 2004–2006: a systematic review and suggestions for improvement. Journal of Thoracic and Cardiovascular Surgery. 2007;134:1128-1135.

    Not in our library10.1016/j.jtcvs.2007.07.021

14. 14.

    Austin PC. A report card on propensity-score matching in the cardiology literature from 2004–2006: results of a systematic review. Circulation: Cardiovascular Quality and Outcomes. 2008;1:62-67.

    Not in our library10.1161/circoutcomes.108.790634

15. 15.

    Austin PC. Some methods of propensity-score matching had superior performance to others: results of an empirical investigation and Monte Carlo simulations. Biometrical Journal. 2009;51:171-184.

    Not in our library10.1002/bimj.200810488

16. 16.

    Austin PC. Type I error rates, coverage of confidence intervals, and variance estimation in propensity-score matched analyses. The International Journal of Biostatistics. 2009;5.

    Not in our library10.2202/1557-4679.1146

17. 17.

    Agresti A, Min Y. Effects and non-effects of paired identical observations in comparing proportions with binary matched–pairs data. Statistics in Medicine. 2004;23:65-75.

    Not in our library10.1002/sim.1589

18. 18.

    Lunceford JK, Davidian M. Stratification and weighting via the propensity score in estimation of causal treatment effects: a comparative study. Statistics in Medicine. 2004;23:2937-2960.

    Not in our library10.1002/sim.1903

19. 19.

    Weitzen S, Lapane KL, Toledano AY, Hume AL, Mor V. Principles for modeling propensity scores in medical research: a systematic literature review. Pharmacoepidemiology and Drug Safety. 2004;13:841-853.

    Not in our library10.1002/pds.969

20. 20.

    Shah BR, Laupacis A, Hux JE, Austin PC. Propensity score methods give similar results to traditional regression modeling in observational studies: a systematic review. Journal of Clinical Epidemiology. 2005;58:550-559.

    Not in our library10.1016/j.jclinepi.2004.10.016

21. 21.

    McCullagh N, Nelder JA. Generalized Linear Models. 1989.

    Not in our library

22. 22.

    Greenland S. Interpretation and choice of effect measures in epidemiologic analyses. American Journal of Epidemiology. 1987;125:761-768.

    Not in our library10.1093/oxfordjournals.aje.a114593

23. 23.

    Austin PC. A data-generation process for data with specified risk differences or numbers needed to treat. Communications in Statistics—Simulation and Computation.

    Not in our library

24. 24.

    Austin PC, Stafford J. The performance of two data-generation processes for data with specified marginal treatment odds ratios. Communications in Statistics—Simulation and Computation. 2008;37:1039-1051.

    Not in our library

25. 25.

    Austin PC, Mamdani MM. A comparison of propensity score methods: a case-study estimating the effectiveness of post-AMI statin use. Statistics in Medicine. 2006;25:2084-2106.

    Not in our library10.1002/sim.2328

26. 26.

    Austin PC, Grootendorst P, Anderson GM. A comparison of the ability of different propensity score models to balance measured variables between treated and untreated subjects: a Monte Carlo study. Statistics in Medicine. 2007;26:734-753.

    Not in our library10.1002/sim.2580

27. 27.

    Lee DS, Austin PC, Rouleau JL, Liu PP, Naimark D, Tu JV. Predicting mortality among patients hospitalized for heart failure: derivation and validation of a clinical model. Journal of the American Medical Association. 2003;290:2581-2587.

    Not in our library10.1001/jama.290.19.2581

28. 28.

    Tu JV, Donovan LR, Lee DS, Austin PC, Ko DT, Wang JT, Newman AM. Quality of cardiac care in Ontario—phase 1. Report 1. 2004.

    Not in our library

29. 29.

    Flury BK, Riedwyl H. Standard distance in univariate and multivariate analysis. The American Statistician. 1986;40:249-251.

    Not in our library

30. 30.

    Austin PC, Manca A, Zwarenstein M, Juurlink DN, Stanbrook MB. A substantial and confusing variation exists in handling of baseline covariates in randomized controlled trials: a review of trials published in leading medical journals. Journal of Clinical Epidemiology. 2010;63:142-153.

    Not in our library10.1016/j.jclinepi.2009.06.002

31. 31.

    Austin PC. The relative ability of different propensity-score methods to balance measured covariates between treated and untreated subjects in observational studies. Medical Decision Making. 2009;29:661-677.

    Not in our library10.1177/0272989x09341755

32. 32.
    Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity‐score matched samples · 2009

    No notice
33. 33.

    Rubin DB. On principles for modeling propensity scores in medical research. Pharmacoepidemiology and Drug Safety. 2004;13:885-887.

    Not in our library10.1002/pds.968

34. 34.

    Imbens GW. Nonparametric estimation of average treatment effects under exogeneity: a review. The Review of Economics and Statistics. 2004;86:4-29.

    Not in our library

Cited by papers we hold 3

• The performance of different propensity score methods for estimating marginal hazard ratios · 2012
  No notice
• Optimal caliper widths for propensity‐score matching when estimating differences in means and differences in proportions in observational studies · 2011
  No notice
• Comparing paired vs non‐paired statistical methods of analyses when making inferences about absolute risk reductions in propensity‐score matched samples · 2011
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.