An investigation of the false discovery rate and the misinterpretation of p -values

Abstract

If you use p =0.05 to suggest that you have made a discovery, you will be wrong at least 30% of the time. If, as is often the case, experiments are underpowered, you will be wrong most of the time. This conclusion is demonstrated from several points of view. First, tree diagrams which show the close analogy with the screening test problem. Similar conclusions are drawn by repeated simulations of t -tests. These mimic what is done in real life, which makes the results more persuasive. The simulation method is used also to evaluate the extent to which effect sizes are over-estimated, especially in underpowered experiments. A script is supplied to allow the reader to do simulations themselves, with numbers appropriate for their own work. It is concluded that if you wish to keep your false discovery rate below 5%, you need to use a three-sigma rule, or to insist on p ≤0.001. And never use the word ‘significant’.

References 21

1. 1.

   Sellke T, Bayarri MJ, Berger JO 2001 Calibration of p values for testing precise null hypotheses. Am. Stat. 55, 62–71. (doi:10.1198/000313001300339950)

   Not in our library

2. 2.
   Why Most Published Research Findings Are False · 2005

   Corrected
3. 3.

   Colquhoun D Lectures on biostatistics. (http://www.dcscience.net/Lectures_on_biostatistics-ocr4.pdf)

   Not in our library

4. 4.

   Scharre DW, Chang SI, Nagaraja HN, Yager-Schweller J, Murden RA 2014 Community cognitive screening using the self-administered gerocognitive examination (SAGE). J. Neuropsychiatry Clin. Neurosci. 26, 369–375. (doi:10.1176/appi.neuropsych.13060145)

   Not in our library10.1176/appi.neuropsych.13060145

5. 5.

   McCartney M 2013 Would doctors routinely asking older patients about their memory improve dementia outcomes?: No. Br. Med. J. 346, 1745 (doi:10.1136/bmj.f1745)

   Not in our library10.1136/bmj.f1745

6. 6.

   Benjamini Y, Hochberg Y 1995 Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. B 57, 289–300.

   Not in our library

7. 7.

   Colquhoun D 2014 Files for running simulated t tests Instructions: http://www.dcscience.net/files/R-script-for-t-tests.pdf; R script: http://www.dcscience.net/files/two_sample-simulation.R; Excel file with results used in paper: http://www.dcscience.net/files/t-test-simulations.xlsx.

   Not in our library

8. 8.

   Cohen D 1962 The Statistical power of abnormal social psychological research—a review. J. Abnorm. Soc. Psychol. 65, 145–153. (doi:10.1037/h0045186)

   Not in our library10.1037/h0045186

9. 9.

   Button KS, Ioannidis JP, Mokrysz C, Nosek BA, Flint J, Robinson ES, Munafo MR 2013 Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376. (doi:10.1038/nrn3475)

   Not in our library10.1038/nrn3475

10. 10.

    Berger JO, Sellke T 1987 Testing a point null hypothesis: the irreconcilability of p-values and evidence. J. Am. Stat. Assoc. 82, 112–122. (doi:10.1080/01621459.1987.10478397)

    Not in our library

11. 11.

    Colquhoun D 2011 Publish-or-perish: Peer review and the corruption of science. See http://www.theguardian.com/science/2011/sep/05/publish-perish-peer-review-science (accessed 1 April 2014)

    Not in our library

12. 12.

    Colquhoun D, Plested AJ 2014 Why you should ignore altmetrics and other bibliometric nightmares. http://www.dcscience.net/?p=6369 (accessed 1 April 2014)

    Not in our library

13. 13.

    Colquhoun D 2013 Problems at Queen Mary University of London. See http://www.dcscience.net/?s=%22Queen+Mary%22 (accessed 1 April 2014)

    Not in our library

14. 14.

    Colquhoun D 2013 Science is harmed by hype. How to live for 969 years. See http://www.dcscience.net/?p=156 (accessed 1 April 2014)

    Not in our library

15. 15.

    Colquhoun D 2013 Acupuncturists show that acupuncture doesn't work, but conclude the opposite: journal fails. See http://www.dcscience.net/?p=4439 (accessed 1 April 2014)

    Not in our library

16. 16.

    Lawrence PA 2007 The mismeasurement of science. Curr. Biol. 17, 583 (doi:10.1016/j.cub.2007.06.014)

    Not in our library10.1016/j.cub.2007.06.014

17. 17.

    Hankins MC 2013 Still not significant. See http://mchankins.wordpress.com/2013/04/21/still-not-significant-2/ (accessed 1 April 2014)

    Not in our library

18. 18.

    Sterne JAC, Davey Smith G 2001 Sifting the evidence—what's wrong with significance tests?. Br. Med. J. 322, 226–231. (doi:10.1136/bmj.322.7280.226)

    Not in our library10.1136/bmj.322.7280.226

19. 19.

    Johnson VE 2013 Revised standards for statistical evidence. Proc. Natl Acad. Sci. USA 110, 19313–19317. (doi:10.1073/pnas.1313476110)

    Not in our library10.1073/pnas.1313476110

20. 20.
    Chapter 11: Genome-Wide Association Studies · 2012

    No notice
21. 21.

    Matthews R 1998 Bayesian Critique of Statistics in Health: The Great Health Hoax. See http://www2.isye.gatech.edu/~brani/isyebayes/bank/pvalue.pdf (accessed 1 April 2014)

    Not in our library

Cited by papers we hold 15

• Automatic Classification Between COVID-19 and Non-COVID-19 Pneumonia Using Symptoms, Comorbidities, and Laboratory Findings: The Khorshid COVID Cohort Study · 2021
  Not checked
• Power contours: Optimising sample size and precision in experimental psychology and human neuroscience. · 2021
  No notice
• Ten common statistical mistakes to watch out for when writing or reviewing a manuscript · 2019
  No notice
• How Many Participants Do We Have to Include in Properly Powered Experiments? A Tutorial of Power Analysis with Reference Tables · 2019
  No notice
• Improving the odds of drug development success through human genomics: modelling study · 2019
  No notice
• Why, When and How to Adjust Your P Values? · 2019
  No notice
• Droplet Digital PCR versus qPCR for gene expression analysis with low abundant targets: from variable nonsense to publication quality data · 2017
  No notice
• The reproducibility of research and the misinterpretation of p -values · 2017
  Corrected
• The earth is flat ( p > 0.05): significance thresholds and the crisis of unreplicable research · 2017
  No notice
• Clinical efficacy and effectiveness of 3D printing: a systematic review · 2017
  No notice
• Editorial January 2016 · 2016
  Not checked
• Recent advances in understanding neuropathic pain: glia, sex differences, and epigenetics [version 1; referees: 2 approved] · 2016
  Not checked
• When Quality Beats Quantity: Decision Theory, Drug Discovery, and the Reproducibility Crisis · 2016
  No notice
• Epigenome-wide Association Studies and the Interpretation of Disease -Omics · 2016
  No notice
• Compensation in Preclinical Huntington's Disease: Evidence From the Track-On HD Study · 2015
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.