Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature

Abstract

We have empirically assessed the distribution of published effect sizes and estimated power by analyzing 26,841 statistical records from 3,801 cognitive neuroscience and psychology papers published recently. The reported median effect size was D = 0.93 (interquartile range: 0.64-1.46) for nominally statistically significant results and D = 0.24 (0.11-0.42) for nonsignificant results. Median power to detect small, medium, and large effects was 0.12, 0.44, and 0.73, reflecting no improvement through the past half-century. This is so because sample sizes have remained small. Assuming similar true effect sizes in both disciplines, power was lower in cognitive neuroscience than in psychology. Journal impact factors negatively correlated with power. Assuming a realistic range of prior probabilities for null hypotheses, false report probability is likely to exceed 50% for the whole literature. In light of our findings, the recently reported low replication success in psychology is realistic, and worse performance may be expected for cognitive neuroscience.

References 62

1. 1.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

2. 2.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

3. 3.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

4. 4.
   Estimating the reproducibility of psychological science · 2015

   No notice
5. 5.
   The prevalence of statistical reporting errors in psychology (1985–2013) · 2015

   No notice
6. 6.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

7. 7.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

8. 8.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

9. 9.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

10. 10.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

11. 11.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

12. 12.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

13. 13.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

14. 14.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

15. 15.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

16. 16.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

17. 17.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

18. 18.
    A Selected Core Microbiome Drives the Early Stages of Three Popular Italian Cheese Manufactures · 2014

    No notice
19. 19.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

20. 20.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

21. 21.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

22. 22.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

23. 23.
    How to Make More Published Research True · 2014

    No notice
24. 24.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

25. 25.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

26. 26.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

27. 27.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

28. 28.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

29. 29.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

30. 30.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

31. 31.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

32. 32.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

33. 33.
    Why Most Published Research Findings Are False · 2005

    Corrected
34. 34.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

35. 35.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

36. 36.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

37. 37.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

38. 38.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

39. 39.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

40. 40.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

Cited by papers we hold 15

• NBS-Predict: A prediction-based extension of the network-based statistic · 2021
  Not checked
• Estimating power in (generalized) linear mixed models: An open introduction and tutorial in R · 2021
  No notice
• Diminished large-scale functional brain networks in absolute pitch during the perception of naturalistic music and audiobooks · 2020
  Not checked
• The Potential of Functional Near-Infrared Spectroscopy-Based Neurofeedback—A Systematic Review and Recommendations for Best Practice · 2020
  Corrected
• Open science challenges, benefits and tips in early career and beyond · 2019
  Corrected
• How Many Participants Do We Have to Include in Properly Powered Experiments? A Tutorial of Power Analysis with Reference Tables · 2019
  No notice
• Estimation for Better Inference in Neuroscience · 2019
  No notice
• Effect Size Guidelines, Sample Size Calculations, and Statistical Power in Gerontology · 2019
  No notice
• Use of the Journal Impact Factor in academic review, promotion, and tenure evaluations · 2019
  No notice
• Indices of Effect Existence and Significance in the Bayesian Framework · 2019
  No notice
• Small sample sizes reduce the replicability of task-based fMRI studies · 2018
  No notice
• Estimation accuracy in the psychological sciences. · 2018
  Not checked
• Cognitive task information is transferred between brain regions via resting-state network topology · 2017
  No notice
• When Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment · 2017
  No notice
• The relation between statistical power and inference in fMRI · 2017
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.