The natural selection of bad science

Abstract

Poor research design and data analysis encourage false-positive findings. Such poor methods persist despite perennial calls for improvement, suggesting that they result from something more than just misunderstanding. The persistence of poor methods results partly from incentives that favour them, leading to the natural selection of bad science. This dynamic requires no conscious strategizing—no deliberate cheating nor loafing—by scientists, only that publication is a principal factor for career advancement. Some normative methods of analysis have almost certainly been selected to further publication instead of discovery. In order to improve the culture of science, a shift must be made away from correcting misunderstandings and towards rewarding understanding. We support this argument with empirical evidence and computational modelling. We first present a 60-year meta-analysis of statistical power in the behavioural sciences and show that power has not improved despite repeated demonstrations of the necessity of increasing power. To demonstrate the logical consequences of structural incentives, we then present a dynamic model of scientific communities in which competing laboratories investigate novel or previously published hypotheses using culturally transmitted research methods. As in the real world, successful labs produce more ‘progeny,’ such that their methods are more often copied and their students are more likely to start labs of their own. Selection for high output leads to poorer methods and increasingly high false discovery rates. We additionally show that replication slows but does not stop the process of methodological deterioration. Improving the quality of research requires change at the institutional level.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 95

1. 1.

   Campbell DT 1976 Assessing the impact of planned social change. Hanover, NH: The Public Affairs Center.

   Not in our library

2. 2.

   Wasserstein RL, Lazar NA 2016 The ASA’s statement on p-values: context, process, and purpose. Am. Stat. 70, 129–133. (doi:10.1080/00031305.2016.1154108)

   Not in our library

3. 3.

   Meehl PE 1967 Theory-testing in psychology and physics: a methodological paradox. Phil. Sci. 34, 103–115. (doi:10.1086/288135)

   Not in our library

4. 4.

   Cohen J 1994 The earth is round (p<.05). Am. Psychol. 49, 997–1003. (doi:10.1037/0003-066X.49.12.997)

   Not in our library

5. 5.

   Enserink M 2012 Final report on Stapel blames field as a whole. Science 338, 1270–1271. (doi:10.1126/science.338.6112.1270)

   Not in our library10.1126/science.338.6112.1270

6. 6.

   Vul E, Harris C, Winkielman P, Pashler H 2009 Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspect. Psychol. Sci. 4, 274–290. (doi:10.1111/j.1745-6924.2009.01125.x)

   Not in our library10.1111/j.1745-6924.2009.01125.x

7. 7.

   Peterson D 2016 The baby factory: difficult research objects, disciplinary standards, and the production of statistical significance. Socius 2, 1–10. (doi:10.1177/2378023115625071)

   Not in our library

8. 8.

   Kerr NL 1998 HARKing: hypothesizing after the results are known. Pers. Soc. Psychol. Rev. 2, 196–217. (doi:10.1207/s15327957pspr0203_4)

   Not in our library10.1207/s15327957pspr0203_4

9. 9.

   John LK, Loewenstein G, Prelec D 2012 Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23, 524–532. (doi:10.1177/0956797611430953)

   Not in our library10.1177/0956797611430953

10. 10.

    Ioannidis JPA 2014 Research accomplishments that are too good to be true. Intensive Care Med. 40, 99–101. (doi:10.1007/s00134-013-3100-z)

    Not in our library10.1007/s00134-013-3100-z

11. 11.

    Simmons JP, Nelson LD, Simonsohn U 2011 False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366. (doi:10.1177/0956797611417632)

    Not in our library10.1177/0956797611417632

12. 12.

    Horton R 2015 Offline: what is medicine’s 5 sigma? Lancet 385, 1380 (doi:10.1016/S0140-6736(15)60696-1)

    Not in our library

13. 13.
    Why Most Published Research Findings Are False · 2005

    Corrected
14. 14.

    Pashler H, Harris CR 2012 Is the replicability crisis overblown? Three arguments examined. Perspect. Psychol. Sci. 7, 531–536. (doi:10.1177/1745691612463401)

    Not in our library10.1177/1745691612463401

15. 15.

    McElreath R, Smaldino PE 2015 Replication, communication, and the population dynamics of scientific discovery. PLoS ONE 10, e0136088 (doi:10.1371/journal.pone.0136088)

    Not in our library10.1371/journal.pone.0136088

16. 16.

    Ioannidis JPA 2005 Contradicted and initially stronger effects in highly cited clinical research. J. Am. Med. Assoc. 294, 218–228. (doi:10.1001/jama.294.2.218)

    Not in our library10.1001/jama.294.2.218

17. 17.

    Ioannidis JPA 2012 Why science is not necessarily self-correcting. Perspect. Psychol. Sci. 7, 645–654. (doi:10.1177/1745691612464056)

    Not in our library10.1177/1745691612464056

18. 18.
    Why Has the Number of Scientific Retractions Increased? · 2013

    Corrected
19. 19.

    Campbell DT 1965 Variation and selective retention in socio-cultural evolution. In Social change in developing areas: a reinterpretation of evolutionary theory (eds H Barringer, G Blanksten, R Mack), pp. 19–49. Cambridge, MA: Schenkman.

    Not in our library

20. 20.

    Skinner BF 1981 Selection by consequences. Science 213, 501–504. (doi:10.1126/science.7244649)

    Not in our library10.1126/science.7244649

21. 21.

    Boyd R, Richerson PJ 1985 Culture and the evolutionary process. Chicago, IL: University of Chicago Press.

    Not in our library

22. 22.

    Mesoudi A 2011 Cultural evolution: how Darwinian theory can explain human culture and synthesize the social sciences. Chicago, IL: University of Chicago Press.

    Not in our library

23. 23.

    Whiten A, Hinde RA, Laland KN, Stringer CB 2011 Culture evolves. Phil. Trans. R. Soc. B 366, 938–948. (doi:10.1098/rstb.2010.0372)

    Not in our library10.1098/rstb.2010.0372

24. 24.

    Smaldino PE 2014 The cultural evolution of emergent group-level traits. Behav. Brain Sci. 37, 243–295. (doi:10.1017/S0140525X13001544)

    Not in our library10.1017/s0140525x13001544

25. 25.

    Acerbi A, Mesoudi A 2015 If we are all cultural Darwinians what’s the fuss about? Clarifying recent disagreements in the field of cultural evolution. Biol. Philos. 30, 481–503. (doi:10.1007/s10539-015-9490-2)

    Not in our library10.1007/s10539-015-9490-2

26. 26.

    Popper K 1979 Objective knowledge: an evolutionary approach. Oxford, UK: Oxford University Press.

    Not in our library

27. 27.

    Hull DL 1988 Science as a process: an evolutionary account of the social and conceptual development of science. Chicago, IL: University of Chicago Press.

    Not in our library

28. 28.

    Larsen PO, von Ins M 2010 The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index. Scientometrics 84, 575–603. (doi:10.1007/s11192-010-0202-z)

    Not in our library10.1007/s11192-010-0202-z

29. 29.

    Bornmann L, Mutz R 2015 Growth rate of modern science: a bibliometric analysis based on the number of publications and cited references. J. Assoc. Inf. Sci. Technol. 66, 2215–2222. (doi:10.1002/asi.23329)

    Not in our library

30. 30.

    Nabout JC, Parreira MR, Teresa FB, Carneiro FM, da Cunha HF, de Souza Ondei L, Caramori SS, Soares TN 2015 Publish (in a group) or perish (alone): the trend from single-to multi-authorship in biological papers. Scientometrics 102, 357–364. (doi:10.1007/s11192-014-1385-5)

    Not in our library

31. 31.

    Wardil L, Hauert C 2015 Cooperation and coauthorship in scientific publication. Phys. Rev. E 91, 012825 (doi:10.1103/PhysRevE.91.012825)

    Not in our library10.1103/physreve.91.012825

32. 32.

    Brischoux F, Angelier F 2015 Academia’s never-ending selection for productivity. Scientometrics 103, 333–336. (doi:10.1007/s11192-015-1534-5)

    Not in our library

33. 33.

    Cyranoski D, Gilbert N, Ledford H, Nayar A, Yahia M 2011 The PhD factory. Nature 472, 276–279. (doi:10.1038/472276a)

    Not in our library10.1038/472276a

34. 34.

    Schillebeeckx M, Maricque B, Lewis C 2013 The missing piece to changing the university culture. Nat. Biotechnol. 31, 938–941. (doi:10.1038/nbt.2706)

    Not in our library10.1038/nbt.2706

35. 35.

    Powell K 2015 The future of the postdoc. Nature 520, 144–147. (doi:10.1038/520144a)

    Not in our library10.1038/520144a

36. 36.

    Kerr E 1995 Too many scientists, too few academic jobs. Nat. Med. 1, 14 (doi:10.1038/nm0195-14)

    Not in our library10.1038/nm0195-14

37. 37.

    Clauset A, Arbesman S, Larremore DB 2015 Systematic inequality and hierarchy in faculty hiring networks. Sci. Adv. 1, e1400005 (doi:10.1126/sciadv.1400005)

    Not in our library10.1126/sciadv.1400005

38. 38.

    Ioannidis JPA, Boyack KW, Klavans R 2014 Estimates of the continuously publishing core in the scientific workforce. PLoS ONE 9, e101698 (doi:10.1371/journal.pone.0101698)

    Not in our library10.1371/journal.pone.0101698

39. 39.

    Vinkers V, Tijdink J, Otte W 2015 Use of positive and negative words in scientific PubMed abstracts between 1974 and 2014: retrospective analysis. Br. Med. J. 351, h6467 (doi:10.1136/bmj.h6467)

    Not in our library10.1136/bmj.h6467

40. 40.

    Hirsch JE 2005 An index to quantify an individual’s scientific research output. Proc. Natl Acad. Sci. USA 102, 16 569–16 572. (doi:10.1073/pnas.0507655102)

    Not in our library10.1073/pnas.0507655102

41. 41.

    Goodhart C 1975 Problems of monetary management: the UK experience. In Papers in monetary economics, volume I. Sydney, Australia: Reserve Bank of Australia.

    Not in our library

42. 42.

    Lucas RE 1976 Econometric policy evaluation: a critique. In Carnegie-Rochester conference series on public policy, vol. 1, pp. 19–46. Amsterdam, The Netherlands: Elsevier.

    Not in our library

43. 43.

    Bartneck C, Kokkelmans S 2011 Detecting h-index manipulation through self-citation analysis. Scientometrics 87, 85–98. (doi:10.1007/s11192-010-0306-5)

    Not in our library10.1007/s11192-010-0306-5

44. 44.

    López-Cózar ED, Robinson-García N, Torres-Salinas D 2014 The Google Scholar experiment: how to index false papers and manipulate bibliometric indicators. J. Assoc. Inf. Sci. Technol. 65, 446–454. (doi:10.1002/asi.23056)

    Not in our library

45. 45.

    Aldhous P 2011 Journal rejects studies contradicting precognition. New Sci. See https://www.newscientist.com/article/dn20447-journal-rejects-studies-contradicting-precognition.

    Not in our library

46. 46.

    Franco A, Malhotra N, Simonovits G 2014 Publication bias in the social sciences: unlocking the file drawer. Science 345, 1502–1505. (doi:10.1126/science.1255484)

    Not in our library10.1126/science.1255484

47. 47.

    Franco A, Simonovits G, Malhotra N 2015 Underreporting in political science survey experiments: comparing questionnaires to published results. Polit. Anal. 23, 306–312. (doi:10.1093/pan/mpv006)

    Not in our library

48. 48.

    Schmidt S 2009 Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Rev. Gen. Psychol. 13, 90–100. (doi:10.1037/a0015108)

    Not in our library

49. 49.

    Makel MC, Plucker JA, Hegarty B 2012 Replications in psychology research: how often do they really occur? Perspect. Psychol. Sci. 7, 537–542. (doi:10.1177/1745691612460688)

    Not in our library10.1177/1745691612460688

50. 50.

    Bissell M 2013 Reproducibility: the risks of the replication drive. Nature 503, 333–334. (doi:10.1038/503333a)

    Not in our library10.1038/503333a

51. 51.

    Schnall S 2014 Clean data: statistical artefacts wash out replication efforts. Soc. Psychol. 45, 315–320. (doi:10.1027/1864-9335/a000204)

    Not in our library

52. 52.

    Kahneman D 2014 A new etiquette for replication. Soc. Psychol. 45, 310–311.

    Not in our library

53. 53.

    Campanario JM 2000 Fraud: retracted articles are still being cited. Nature 408, 288 (doi:10.1038/35042753)

    Not in our library10.1038/35042753

54. 54.

    Aschwanden C 2015 Science isn’t broken. FiveThirtyEight. See http://fivethirtyeight.com/features/science-isnt-broken/.

    Not in our library

55. 55.

    Bennett CM, Miller M, Wolford G 2009 Neural correlates of interspecies perspective taking in the post-mortem Atlantic salmon: an argument for multiple comparisons correction. Neuroimage 47, S125 (doi:10.1016/S1053-8119(09)71202-9)

    Not in our library

56. 56.

    Wagenmakers E-J, Wetzels R, Borsboom D, van der Maas HL 2011 Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011). J. Pers. Soc. Psychol. 100, 426–432. (doi:10.1037/a0022790)

    Not in our library10.1037/a0022790

57. 57.

    Gigerenzer G 1998 Surrogates for theories. Theory Psychol. 8, 195–204. (doi:10.1177/0959354398082006)

    Not in our library

58. 58.

    Gelman A, Loken E 2014 The statistical crisis in science. Am. Sci. 102, 460–465. (doi:10.1511/2014.111.460)

    Not in our library

59. 59.

    Cohen J 1992 Statistical power analysis. Curr. Dir. Psychol. Sci. 1, 98–101. (doi:10.1111/1467-8721.ep10768783)

    Not in our library

60. 60.

    Richard FD, Bond CF, Stokes-Zoota JJ 2003 One hundred years of social psychology quantitatively described. Rev. Gen. Psychol. 7, 331–363. (doi:10.1037/1089-2680.7.4.331)

    Not in our library

61. 61.

    Aguinis H, Beaty JC, Boik RJ, Pierce CA 2005 Effect size and power in assessing moderating effects of categorical variables using multiple regression: a 30-year review. J. Appl. Psychol. 90, 94–107. (doi:10.1037/0021-9010.90.1.94)

    Not in our library10.1037/0021-9010.90.1.94

62. 62.
    Publication Bias in Psychology: A Diagnosis Based on the Correlation between Effect Size and Sample Size · 2014

    No notice
63. 63.

    Button KS, Ioannidis JPA, Mokrysz C, Nosek BA, Flint J, Robinson ESJ, Munafò MR 2013 Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376. (doi:10.1038/nrn3475)

    Not in our library10.1038/nrn3475

64. 64.

    Lakens D, Evers ERK 2014 Sailing from the seas of chaos into the corridor of stability: practical recommendations to increase the informational value of studies. Perspect. Psychol. Sci. 9, 278–292. (doi:10.1177/1745691614528520)

    Not in our library10.1177/1745691614528520

65. 65.

    Cohen J 1962 The statistical power of abnormal-social psychological research: a review. J. Abnorm. Soc. Psychol. 65, 145–153. (doi:10.1037/h0045186)

    Not in our library10.1037/h0045186

66. 66.

    Sedlmeier P, Gigerenzer G 1989 Do studies of statistical power have an effect on the power of studies? Psychol. Bull. 105, 309–316. (doi:10.1037/0033-2909.105.2.309)

    Not in our library

67. 67.

    Rossi JS 1990 Statistical power of psychological research: what have we gained in 20 years? J. Consult. Clin. Psychol. 58, 646–656. (doi:10.1037/0022-006X.58.5.646)

    Not in our library10.1037//0022-006x.58.5.646

68. 68.

    Vankov I, Bowers J, Munafò MR 2014 On the persistence of low power in psychological science. Q. J. Exp. Psychol. 67, 1037–1040. (doi:10.1080/17470218.2014.885986)

    Not in our library10.1080/17470218.2014.885986

69. 69.

    Nosek BA, Spies JR, Motyl M 2012 Scientific utopia II. Restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7, 615–631. (doi:10.1177/1745691612459058)

    Not in our library10.1177/1745691612459058

70. 70.

    Smaldino PE, Calanchini J, Pickett CL 2015 Theory development with agent-based models. Organ. Psychol. Rev. 5, 300–317. (doi:10.1177/2041386614546944)

    Not in our library

71. 71.

    Wimsatt WC 1987 False models as means to truer theories. In Neutral models in biology (eds M Nitecki, A Hoffman), pp. 23–55. Oxford, UK: Oxford University Press.

    Not in our library

72. 72.

    Epstein JM 2008 Why model? J. Artif. Soc. Soc. Simul. 11, 12.

    Not in our library

73. 73.

    MacMillan NA, Creelman CD 1991 Detection theory: a user’s guide. Cambridge, UK: Cambridge University Press.

    Not in our library

74. 74.

    Vines TH et al. 2014 The availability of research data declines rapidly with article age. Curr. Biol. 24, 94–97. (doi:10.1016/j.cub.2013.11.014)

    Not in our library10.1016/j.cub.2013.11.014

75. 75.

    MacArthur D 2012 Face up to false positives. Nature 487, 427–428. (doi:10.1038/487427a)

    Not in our library10.1038/487427a

76. 76.
    The Assessment of Science: The Relative Merits of Post-Publication Review, the Impact Factor, and the Number of Citations · 2013

    No notice
77. 77.

    Lu SF, Jin GZ, Uzzi B, Jones B 2013 The retraction penalty: evidence from the web of science. Sci. Rep. 3, 3146 (doi:10.1038/srep03146)

    Not in our library10.1038/srep03146

78. 78.

    Rosenblatt M 2016 An incentive-based approach for improving data reproducibility. Sci. Transl. Med. 8, 336ed5 (doi:10.1126/scitranslmed.aaf5003)

    Not in our library10.1126/scitranslmed.aaf5003

79. 79.

    Frank RH 2011 The Darwin economy: liberty, competition, and the common good. Princeton, NJ: Princeton University Press.

    Not in our library

80. 80.

    Wilson DS, Hayes SC, Biglan A, Embry DD 2014 Evolving the future: toward a science of intentional change. Behav. Brain Sci. 37, 395–460. (doi:10.1017/S0140525X13001593)

    Not in our library10.1017/s0140525x13001593

81. 81.

    Nosek BA, Lakens D 2014 Registered reports: a method to increase the credibility of published results. Soc. Psychol. 45, 137–141. (doi:10.1027/1864-9335/a000192)

    Not in our library

82. 82.
    How to Make More Published Research True · 2014

    No notice
83. 83.

    Fischer J, Ritchie EG, Hanspach J 2012 An academia beyond quantity: a reply to Loyola et al. and Halme et al. Trends Ecol. Evol. 27, 587–588. (doi:10.1016/j.tree.2012.08.009)

    Not in our library

84. 84.
    Deep impact: unintended consequences of journal rank · 2013

    No notice
85. 85.

    Begley CG, Buchan AM, Dirnagl U 2015 Institutions must do their part for reproducibility. Nature 525, 25–27. (doi:10.1038/525025a)

    Not in our library10.1038/525025a

86. 86.

    MacCoun R, Perlmutter S 2015 Hide results to seek the truth. Nature 526, 187–189. (doi:10.1038/526187a)

    Not in our library10.1038/526187a

87. 87.

    Gigerenzer G, Marewski JN 2015 Surrogate science: the idol of a universal method for scientific inference. J. Manag. 41, 421–440. (doi:10.1177/0149206314547522)

    Not in our library

88. 88.

    Sills J 2016 Measures of success. Science 352, 28–30. (doi:10.1126/science.352.6281.28)

    Not in our library

89. 89.

    Sarewitz D 2016 The pressure to publish pushes down quality. Nature 533, 147 (doi:10.1038/533147a)

    Not in our library10.1038/533147a

90. 90.

    Rand DG, Nowak MA 2013 Human cooperation. Trends Cogn. Sci. 17, 413–425. (doi:10.1016/j.tics.2013.06.003)

    Not in our library10.1016/j.tics.2013.06.003

91. 91.

    North DC 1990 Institutions, institutional change and economic performance. Cambridge, UK: Cambridge University Press.

    Not in our library

92. 92.

    Aoki M 2007 Endogenizing institutions and institutional changes. J. Inst. Econ. 3, 1–31. (doi:10.1017/S1744137406000531)

    Not in our library

93. 93.

    Martin BR 2016 Editors’ JIF-boosting strategems: which are appropriate and which not? Res. Policy 45, 1–7. (doi:10.1016/j.respol.2015.09.001)

    Not in our library

94. 94.
    Concentration of the most-cited papers in the scientific literature: analysis of journal ecosystems. · 2006

    Not checked
95. 95.

    Fang FC, Casadevall A 2011 Retracted science and the retraction index. Infect. Immun. 79, 3855–3859. (doi:10.1128/IAI.05661-11)

    Not in our library10.1128/iai.05661-11

Cited by papers we hold 11

• Challenges for assessing replicability in preclinical cancer biology · 2021
  No notice
• Open science and modified funding lotteries can impede the natural selection of bad science · 2019
  Not checked
• On the prevalence of uninformative parameters in statistical models applying model selection in applied ecology. · 2019
  Not checked
• How Many Participants Do We Have to Include in Properly Powered Experiments? A Tutorial of Power Analysis with Reference Tables · 2019
  No notice
• Scientific productivity: An exploratory study of metrics and incentives. · 2018
  Not checked
• Power Analysis and Effect Size in Mixed Effects Models: A Tutorial · 2018
  No notice
• Questionable research practices in ecology and evolution · 2018
  No notice
• Quality Uncertainty Erodes Trust in Science · 2017
  Not checked
• The reproducibility of research and the misinterpretation of p -values · 2017
  Corrected
• The earth is flat ( p > 0.05): significance thresholds and the crisis of unreplicable research · 2017
  No notice
• A manifesto for reproducible science · 2017
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.