Comparing paired vs non‐paired statistical methods of analyses when making inferences about absolute risk reductions in propensity‐score matched samples

Abstract

Propensity‐score matching allows one to reduce the effects of treatment‐selection bias or confounding when estimating the effects of treatments when using observational data. Some authors have suggested that methods of inference appropriate for independent samples can be used for assessing the statistical significance of treatment effects when using propensity‐score matching. Indeed, many authors in the applied medical literature use methods for independent samples when making inferences about treatment effects using propensity‐score matched samples. Dichotomous outcomes are common in healthcare research. In this study, we used Monte Carlo simulations to examine the effect on inferences about risk differences (or absolute risk reductions) when statistical methods for independent samples are used compared with when statistical methods for paired samples are used in propensity‐score matched samples. We found that compared with using methods for independent samples, the use of methods for paired samples resulted in: (i) empirical type I error rates that were closer to the advertised rate; (ii) empirical coverage rates of 95 per cent confidence intervals that were closer to the advertised rate; (iii) narrower 95 per cent confidence intervals; and (iv) estimated standard errors that more closely reflected the sampling variability of the estimated risk difference. Differences between the empirical and advertised performance of methods for independent samples were greater when the treatment‐selection process was stronger compared with when treatment‐selection process was weaker. We recommend using statistical methods for paired samples when using propensity‐score matched samples for making inferences on the effect of treatment on the reduction in the probability of an event occurring. Copyright © 2011 John Wiley & Sons, Ltd.

Shown for reading. Its publisher allows quoting it, not republishing it.

References 18

1. 1.

   Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70:41-55.

   Not in our library

2. 2.

   Austin PC. The relative ability of different propensity-score methods to balance measured covariates between treated and untreated subjects in observational studies. Medical Decision Making. 2009;29:661-677.

   Not in our library10.1177/0272989x09341755

3. 3.

   Schafer JL, Kang J. Average causal effects from nonrandomized studies: a practical guide and simulated example. Psychological Methods. 2008;13:279-313.

   Not in our library10.1037/a0014268

4. 4.

   Austin PC. Propensity-score matching in the cardiovascular surgery literature from 2004 to 2006: a systematic review and suggestions for improvement. Journal of Thoracic and Cardiovascular Surgery. 2007;134:1128-1135.

   Not in our library10.1016/j.jtcvs.2007.07.021

5. 5.

   Austin PC. A critical appraisal of propensity score matching in the medical literature from 1996 to 2003. Statistics in Medicine. 2008;27:2037-2049.

   Not in our library10.1002/sim.3150

6. 6.

   Austin PC. A report card on propensity-score matching in the cardiology literature from 2004 to 2006: results of a systematic review. Circulation: Cardiovascular Quality and Outcomes. 2008;1:62-67.

   Not in our library10.1161/circoutcomes.108.790634

7. 7.

   Shah BR, Laupacis A, Hux JE, Austin PC. Propensity score methods give similar results to traditional regression modeling in observational studies: a systematic review. Journal of Clinical Epidemiology. 2005;58:550-559.

   Not in our library10.1016/j.jclinepi.2004.10.016

8. 8.

   Austin PC, Manca A, Zwarenstein M, Juurlink DN, Stanbrook MB. A substantial and confusing variation exists in handling of baseline covariates in randomized controlled trials: a review of trials published in leading medical journals. Journal of Clinical Epidemiology. 2010;63:142-153.

   Not in our library10.1016/j.jclinepi.2009.06.002

9. 9.
   The performance of different propensity-score methods for estimating differences in proportions (risk differences or absolute risk reductions) in observational studies · 2010

   No notice
10. 10.
    Optimal caliper widths for propensity‐score matching when estimating differences in means and differences in proportions in observational studies · 2011

    No notice
11. 11.

    Imbens GW. Nonparametric estimation of average treatment effects under exogeneity: a review. The Review of Economics and Statistics. 2004;86:4-29.

    Not in our library

12. 12.

    Austin PC. A data-generation process for data with specified risk differences or numbers needed to treat. Communications in Statistics – Simulation and Computation. 2010;39:563-577.

    Not in our library

13. 13.

    Agresti A. An Introduction to Categorical Data Analysis. 1996.

    Not in our library

14. 14.

    Agresti A, Min Y. Effects and non-effects of paired identical observations in comparing proportions with binary matched-pairs data. Statistics in Medicine. 2004;23:65-75.

    Not in our library10.1002/sim.1589

15. 15.

    Austin PC. Type I error rates, coverage of confidence intervals, and variance estimation in propensity-score matched analyses. The International Journal of Biostatistics. 2009;5.

    Not in our library10.2202/1557-4679.1146

16. 16.

    Austin PC, Grootendorst P, Normand SLT, Anderson GM. Conditioning on the propensity score can result in biased estimation of common measures of treatment effect: a Monte Carlo study. Statistics in Medicine. 2007;26:754-768.

    Not in our library10.1002/sim.2618

17. 17.

    Austin PC. The performance of different propensity score methods for estimating marginal odds ratios. Statistics in Medicine. 2007;26:3078-3094.

    Not in our library10.1002/sim.2781

18. 18.

    Austin PC. The performance of different propensity score methods for estimating relative risks. Journal of Clinical Epidemiology. 2008;61:537-545.

    Not in our library10.1016/j.jclinepi.2007.07.011

Cited by papers we hold 3

• Incidence and outcomes of hospitalization for community-acquired, ventilator-associated and non-ventilator hospital-acquired pneumonias in patients with type 2 diabetes mellitus in Spain · 2020
  Not checked
• Propensity score matching in otolaryngologic literature: A systematic review and critical appraisal. · 2020
  Not checked
• The performance of different propensity score methods for estimating marginal hazard ratios · 2012
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.