The natural selection of bad science

Abstract

Poor research design and data analysis encourage false-positive findings. Such poor methods persist despite perennial calls for improvement, suggesting that they result from something more than just misunderstanding. The persistence of poor methods results partly from incentives that favour them, leading to the natural selection of bad science. This dynamic requires no conscious strategizing—no deliberate cheating nor loafing—by scientists, only that publication is a principal factor for career advancement. Some normative methods of analysis have almost certainly been selected to further publication instead of discovery. In order to improve the culture of science, a shift must be made away from correcting misunderstandings and towards rewarding understanding. We support this argument with empirical evidence and computational modelling. We first present a 60-year meta-analysis of statistical power in the behavioural sciences and show that power has not improved despite repeated demonstrations of the necessity of increasing power. To demonstrate the logical consequences of structural incentives, we then present a dynamic model of scientific communities in which competing laboratories investigate novel or previously published hypotheses using culturally transmitted research methods. As in the real world, successful labs produce more ‘progeny,’ such that their methods are more often copied and their students are more likely to start labs of their own. Selection for high output leads to poorer methods and increasingly high false discovery rates. We additionally show that replication slows but does not stop the process of methodological deterioration. Improving the quality of research requires change at the institutional level.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 95

1. 1.

   Campbell DT 1976 Assessing the impact of planned social change. Hanover, NH: The Public Affairs Center.

   Not in our library

2. 2.

   Wasserstein RL, Lazar NA 2016 The ASA’s statement on p-values: context, process, and purpose. Am. Stat. 70, 129–133. (doi:10.1080/00031305.2016.1154108)

   Not in our library

3. 3.

   Meehl PE 1967 Theory-testing in psychology and physics: a methodological paradox. Phil. Sci. 34, 103–115. (doi:10.1086/288135)

   Not in our library

4. 4.

   Cohen J 1994 The earth is round (p<.05). Am. Psychol. 49, 997–1003. (doi:10.1037/0003-066X.49.12.997)

   Not in our library

5. 5.

   Enserink M 2012 Final report on Stapel blames field as a whole. Science 338, 1270–1271. (doi:10.1126/science.338.6112.1270)

   Not in our library10.1126/science.338.6112.1270

6. 6.

   Vul E, Harris C, Winkielman P, Pashler H 2009 Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspect. Psychol. Sci. 4, 274–290. (doi:10.1111/j.1745-6924.2009.01125.x)

   Not in our library10.1111/j.1745-6924.2009.01125.x

7. 7.

   Peterson D 2016 The baby factory: difficult research objects, disciplinary standards, and the production of statistical significance. Socius 2, 1–10. (doi:10.1177/2378023115625071)

   Not in our library

8. 8.

   Kerr NL 1998 HARKing: hypothesizing after the results are known. Pers. Soc. Psychol. Rev. 2, 196–217. (doi:10.1207/s15327957pspr0203_4)

   Not in our library10.1207/s15327957pspr0203_4

9. 9.

   John LK, Loewenstein G, Prelec D 2012 Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23, 524–532. (doi:10.1177/0956797611430953)

   Not in our library10.1177/0956797611430953

10. 10.

    Ioannidis JPA 2014 Research accomplishments that are too good to be true. Intensive Care Med. 40, 99–101. (doi:10.1007/s00134-013-3100-z)

    Not in our library10.1007/s00134-013-3100-z

11. 11.

    Simmons JP, Nelson LD, Simonsohn U 2011 False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366. (doi:10.1177/0956797611417632)

    Not in our library10.1177/0956797611417632

12. 12.

    Horton R 2015 Offline: what is medicine’s 5 sigma? Lancet 385, 1380 (doi:10.1016/S0140-6736(15)60696-1)

    Not in our library

13. 13.
    Why Most Published Research Findings Are False · 2005

    Corrected
14. 14.

    Pashler H, Harris CR 2012 Is the replicability crisis overblown? Three arguments examined. Perspect. Psychol. Sci. 7, 531–536. (doi:10.1177/1745691612463401)

    Not in our library10.1177/1745691612463401

15. 15.

    McElreath R, Smaldino PE 2015 Replication, communication, and the population dynamics of scientific discovery. PLoS ONE 10, e0136088 (doi:10.1371/journal.pone.0136088)

    Not in our library10.1371/journal.pone.0136088

16. 16.

    Ioannidis JPA 2005 Contradicted and initially stronger effects in highly cited clinical research. J. Am. Med. Assoc. 294, 218–228. (doi:10.1001/jama.294.2.218)

    Not in our library10.1001/jama.294.2.218

17. 17.

    Ioannidis JPA 2012 Why science is not necessarily self-correcting. Perspect. Psychol. Sci. 7, 645–654. (doi:10.1177/1745691612464056)

    Not in our library10.1177/1745691612464056

18. 18.
    Why Has the Number of Scientific Retractions Increased? · 2013

    Corrected
19. 19.

    Campbell DT 1965 Variation and selective retention in socio-cultural evolution. In Social change in developing areas: a reinterpretation of evolutionary theory (eds H Barringer, G Blanksten, R Mack), pp. 19–49. Cambridge, MA: Schenkman.

    Not in our library

20. 20.

    Skinner BF 1981 Selection by consequences. Science 213, 501–504. (doi:10.1126/science.7244649)

    Not in our library10.1126/science.7244649

21. 21.

    Boyd R, Richerson PJ 1985 Culture and the evolutionary process. Chicago, IL: University of Chicago Press.

    Not in our library

22. 22.

    Mesoudi A 2011 Cultural evolution: how Darwinian theory can explain human culture and synthesize the social sciences. Chicago, IL: University of Chicago Press.

    Not in our library

23. 23.

    Whiten A, Hinde RA, Laland KN, Stringer CB 2011 Culture evolves. Phil. Trans. R. Soc. B 366, 938–948. (doi:10.1098/rstb.2010.0372)

    Not in our library10.1098/rstb.2010.0372

24. 24.

    Smaldino PE 2014 The cultural evolution of emergent group-level traits. Behav. Brain Sci. 37, 243–295. (doi:10.1017/S0140525X13001544)

    Not in our library10.1017/s0140525x13001544

25. 25.

    Acerbi A, Mesoudi A 2015 If we are all cultural Darwinians what’s the fuss about? Clarifying recent disagreements in the field of cultural evolution. Biol. Philos. 30, 481–503. (doi:10.1007/s10539-015-9490-2)

    Not in our library10.1007/s10539-015-9490-2

26. 26.

    Popper K 1979 Objective knowledge: an evolutionary approach. Oxford, UK: Oxford University Press.

    Not in our library

27. 27.

    Hull DL 1988 Science as a process: an evolutionary account of the social and conceptual development of science. Chicago, IL: University of Chicago Press.

    Not in our library

28. 28.

    Larsen PO, von Ins M 2010 The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index. Scientometrics 84, 575–603. (doi:10.1007/s11192-010-0202-z)

    Not in our library10.1007/s11192-010-0202-z

29. 29.

    Bornmann L, Mutz R 2015 Growth rate of modern science: a bibliometric analysis based on the number of publications and cited references. J. Assoc. Inf. Sci. Technol. 66, 2215–2222. (doi:10.1002/asi.23329)

    Not in our library

30. 30.

    Nabout JC, Parreira MR, Teresa FB, Carneiro FM, da Cunha HF, de Souza Ondei L, Caramori SS, Soares TN 2015 Publish (in a group) or perish (alone): the trend from single-to multi-authorship in biological papers. Scientometrics 102, 357–364. (doi:10.1007/s11192-014-1385-5)

    Not in our library

31. 31.

    Wardil L, Hauert C 2015 Cooperation and coauthorship in scientific publication. Phys. Rev. E 91, 012825 (doi:10.1103/PhysRevE.91.012825)

    Not in our library10.1103/physreve.91.012825

32. 32.

    Brischoux F, Angelier F 2015 Academia’s never-ending selection for productivity. Scientometrics 103, 333–336. (doi:10.1007/s11192-015-1534-5)

    Not in our library

33. 33.

    Cyranoski D, Gilbert N, Ledford H, Nayar A, Yahia M 2011 The PhD factory. Nature 472, 276–279. (doi:10.1038/472276a)

    Not in our library10.1038/472276a

34. 34.

    Schillebeeckx M, Maricque B, Lewis C 2013 The missing piece to changing the university culture. Nat. Biotechnol. 31, 938–941. (doi:10.1038/nbt.2706)

    Not in our library10.1038/nbt.2706

35. 35.

    Powell K 2015 The future of the postdoc. Nature 520, 144–147. (doi:10.1038/520144a)

    Not in our library10.1038/520144a

36. 36.

    Kerr E 1995 Too many scientists, too few academic jobs. Nat. Med. 1, 14 (doi:10.1038/nm0195-14)

    Not in our library10.1038/nm0195-14

37. 37.

    Clauset A, Arbesman S, Larremore DB 2015 Systematic inequality and hierarchy in faculty hiring networks. Sci. Adv. 1, e1400005 (doi:10.1126/sciadv.1400005)

    Not in our library10.1126/sciadv.1400005

38. 38.

    Ioannidis JPA, Boyack KW, Klavans R 2014 Estimates of the continuously publishing core in the scientific workforce. PLoS ONE 9, e101698 (doi:10.1371/journal.pone.0101698)

    Not in our library10.1371/journal.pone.0101698

39. 39.

    Vinkers V, Tijdink J, Otte W 2015 Use of positive and negative words in scientific PubMed abstracts between 1974 and 2014: retrospective analysis. Br. Med. J. 351, h6467 (doi:10.1136/bmj.h6467)

    Not in our library10.1136/bmj.h6467

40. 40.

    Hirsch JE 2005 An index to quantify an individual’s scientific research output. Proc. Natl Acad. Sci. USA 102, 16 569–16 572. (doi:10.1073/pnas.0507655102)

    Not in our library10.1073/pnas.0507655102

Cited by papers we hold 11

• Challenges for assessing replicability in preclinical cancer biology · 2021
  No notice
• Open science and modified funding lotteries can impede the natural selection of bad science · 2019
  Not checked
• On the prevalence of uninformative parameters in statistical models applying model selection in applied ecology. · 2019
  Not checked
• How Many Participants Do We Have to Include in Properly Powered Experiments? A Tutorial of Power Analysis with Reference Tables · 2019
  No notice
• Scientific productivity: An exploratory study of metrics and incentives. · 2018
  Not checked
• Power Analysis and Effect Size in Mixed Effects Models: A Tutorial · 2018
  No notice
• Questionable research practices in ecology and evolution · 2018
  No notice
• Quality Uncertainty Erodes Trust in Science · 2017
  Not checked
• The reproducibility of research and the misinterpretation of p -values · 2017
  Corrected
• The earth is flat ( p > 0.05): significance thresholds and the crisis of unreplicable research · 2017
  No notice
• A manifesto for reproducible science · 2017
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.