Publication bias and the canonization of false facts

Abstract

Science is facing a “replication crisis” in which many experimental findings cannot be replicated and are likely to be false. Does this imply that many scientific facts are false as well? To find out, we explore the process by which a claim becomes fact. We model the community’s confidence in a claim as a Markov process with successive published results shifting the degree of belief. Publication bias in favor of positive findings influences the distribution of published results. We find that unless a sufficient fraction of negative results are published, false claims frequently can become canonized as fact. Data-dredging, p-hacking, and similar behaviors exacerbate the problem. Should negative results become easier to publish as a claim approaches acceptance as a fact, however, true and false claims would be more readily distinguished. To the degree that the model reflects the real world, there may be serious concerns about the validity of purported facts in some disciplines.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 55

1. 1.

   Arbesman S. The Half-Life of Facts: Why Everything We Know Has an Expiration Date. 2012.

   Not in our library

2. 2.

   Baker M. 1,500 scientists lift the lid on reproducibility. Nature. 2016;533:452-454.

   Not in our library10.1038/533452a

3. 3.

   Begg CB, Berlin JA. Publication bias: a problem in interpreting medical data. Journal of the Royal Statistical Society. Series A. 1988;151:419-463.

   Not in our library10.2307/2982993

4. 4.

   Begley CG, Ellis LM. Drug development: Raise standards for preclinical cancer research. Nature. 2012;483:531-533.

   Not in our library10.1038/483531a

5. 5.

   Bernstein E, Caudy AA, Hammond SM, Hannon GJ. Role for a bidentate ribonuclease in the initiation step of RNA interference. Nature. 2001;409:363-366.

   Not in our library10.1038/35053110

6. 6.

   Camerer CF, Dreber A, Forsell E, Ho TH, Huber J, Johannesson M, Kirchler M, Almenberg J, Altmejd A, Chan T, Heikensten E, Holzmeister F, Imai T, Isaksson S, Nave G, Pfeiffer T, Razen M, Wu H. Evaluating replicability of laboratory experiments in economics. Science. 2016;351:1433-1436.

   Not in our library10.1126/science.aaf0918

7. 7.

   Chan AW, Altman DG. Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ. 2005;330:753.

   Not in our library10.1136/bmj.38356.424606.8f

8. 8.

   Chang AC, Li P. Is economics research replicable? sixty published papers from thirteen journals say “usually Not. Finance and Economics Discussion Series. 2015;083:1-26.

   Not in our library10.17016/feds.2015.083

9. 9.

   Csada RD, James PC, Espie RHM. The "file drawer problem" of non-significant results: does it apply to biological research? Oikos. 1996;76:591-593.

   Not in our library10.2307/3546355

10. 10.

    de Winter J, Happee R. Why selective publication of statistically significant results can be effective. PLoS One. 2013;8:e66463.

    Not in our library10.1371/journal.pone.0066463

11. 11.

    Dickersin K, Min YI, Meinert CL. Factors influencing publication of research results. follow-up of applications submitted to two institutional review boards. JAMA. 1992;267:374-378.

    Not in our library10.1001/jama.267.3.374

12. 12.

    Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA. 1990;263:1385.

    Not in our library10.1001/jama.1990.03440100097014

13. 13.

    Easterbrook PJ, Gopalan R, Berlin JA, Matthews DR. Publication bias in clinical research. The Lancet. 1991;337:867-872.

    Not in our library10.1016/0140-6736(91)90201-y

14. 14.

    Ebrahim S, Sohani ZN, Montoya L, Agarwal A, Thorlund K, Mills EJ, Ioannidis JP. Reanalyses of randomized clinical trial data. JAMA. 2014;312:1024-1032.

    Not in our library10.1001/jama.2014.9646

15. 15.

    Editors. Go forth and replicate! Nature. 2016;536:373.

    Not in our library10.1038/536373a

16. 16.

    Eklund A, Nichols TE, Knutsson H. Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates. PNAS. 2016;113:7900-7905.

    Not in our library10.1073/pnas.1602413113

17. 17.
    An open investigation of the reproducibility of cancer biology research · 2014

    No notice
18. 18.

    Fanelli D. Negative results are disappearing from most disciplines and countries. Scientometrics. 2012;90:891-904.

    Not in our library10.1007/s11192-011-0494-7

19. 19.

    Franco A, Malhotra N, Simonovits G. Social science. publication bias in the social sciences: unlocking the file drawer. Science. 2014;345:1502-1505.

    Not in our library10.1126/science.1255484

20. 20.

    Gelman A, Loken E. The statistical crisis in science. American Scientist. 2014;102:460-465.

    Not in our library10.1511/2014.111.460

21. 21.

    Good IJ. On the principle of total evidence. The British Journal for the Philosophy of Science. 1967;17:319-321.

    Not in our library10.1093/bjps/17.4.319

22. 22.

    Higginson AD, Munafò MR. Current incentives for scientists lead to underpowered studies with erroneous conclusions. PLoS Biology. 2016;14:e2000995.

    Not in our library10.1371/journal.pbio.2000995

23. 23.
    Why Most Published Research Findings Are False · 2005

    Corrected
24. 24.

    Ioannidis JP, Trikalinos TA. Early extreme contradictory estimates may appear in published research: the Proteus phenomenon in molecular genetics research and randomized trials. Journal of Clinical Epidemiology. 2005;58:543-549.

    Not in our library10.1016/j.jclinepi.2004.10.019

25. 25.

    Jaskiewicz L, Filipowicz W. Role of dicer in posttranscriptional RNA silencing. Current Topics in Microbiology and Immunology. 2008;320:77-97.

    Not in our library10.1007/978-3-540-75157-1_4

26. 26.

    Jennions MD, Moller AP. Relationships fade with time: a meta-analysis of temporal trends in publication in ecology and evolution. Proceedings of the Royal Society B: Biological Sciences. 2002;269:43-48.

    Not in our library10.1098/rspb.2001.1832

27. 27.

    Kasenda B, von Elm E, You JJ, Blümle A, Tomonaga Y, Saccilotto R, Amstutz A, Bengough T, Meerpohl JJ, Stegert M, Olu KK, Tikkinen KA, Neumann I, Carrasco-Labra A, Faulhaber M, Mulla SM, Mertz D, Akl EA, Bassler D, Busse JW, Ferreira-González I, Lamontagne F, Nordmann A, Gloy V, Raatz H, Moja L, Ebrahim S, Schandelmaier S, Sun X, Vandvik PO, Johnston BC, Walter MA, Burnand B, Schwenkglenks M, Hemkens LG, Bucher HC, Guyatt GH, Briel M, Blümle A. Agreements between industry and academia on publication rights: A retrospective study of protocols and publications of randomized clinical trials. PLoS Medicine. 2016;13:e1002046.

    Not in our library10.1371/journal.pmed.1002046

28. 28.

    Kerr NL. HARKing: hypothesizing after the results are known. Personality and Social Psychology Review. 1998;2:196-217.

    Not in our library10.1207/s15327957pspr0203_4

29. 29.

    Knight J. Negative results: Null and void. Nature. 2003;422:554-555.

    Not in our library10.1038/422554a

30. 30.

    Latour B. Science in Action: How to Follow Scientists and Engineers Through Society. 1987.

    Not in our library

31. 31.

    Matosin N, Frank E, Engel M, Lum JS, Newell KA. Negativity towards negative results: a discussion of the disconnect between scientific worth and scientific culture. Disease Models & Mechanisms. 2014;7:171-173.

    Not in our library10.1242/dmm.015123

32. 32.

    McElreath R, Smaldino PE. Replication, communication, and the population dynamics of scientific discovery. PLoS One. 2015;10:e0136088.

    Not in our library10.1371/journal.pone.0136088

33. 33.
    The Extent and Consequences of P-Hacking in Science · 2015

    No notice
34. 34.

    Nelson LD, simmons JP, simonsohn U. let’s publish fewer papers. Psychological Inquiry. 2012;233:291-293.

    Not in our library10.1080/1047840x.2012.705245

35. 35.

    Newcombe RG. Towards a reduction in publication bias. BMJ. 1987;295:656-659.

    Not in our library10.1136/bmj.295.6599.656

36. 36.

    Norris JR. Markov Chains. 1998.

    Not in our library

37. 37.

    Le Noury J, Nardo JM, Healy D, Jureidini J, Raven M, Tufanaru C, Abi-Jaoude E. Restoring study 329: efficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence. BMJ. 2015;351:h4320.

    Not in our library10.1136/bmj.h4320

38. 38.

    Olson CM, Rennie D, Cook D, Dickersin K, Flanagin A, Hogan JW, Zhu Q, Reiling J, Pace B. Publication bias in editorial decision making. JAMA. 2002;287:2825-2828.

    Not in our library10.1001/jama.287.21.2825

39. 39.
    Estimating the reproducibility of psychological science · 2015

    No notice
40. 40.

    Palmer AR. Quasi-replication and the contract of error: lessons from sex ratios, heritabilities and fluctuating asymmetry. Annual Review of Ecology and Systematics. 2000;31:441-480.

    Not in our library10.1146/annurev.ecolsys.31.1.441

41. 41.

    Pocock SJ, Hughes MD, Lee RJ. Statistical problems in the reporting of clinical trials. New England Journal of Medicine. 1987;317:426-432.

    Not in our library10.1056/nejm198708133170706

42. 42.

    Poulin R. Manipulation of host behaviour by parasites: a weakening paradigm? Proceedings of the Royal Society B: Biological Sciences. 2000;267:787-792.

    Not in our library10.1098/rspb.2000.1072

43. 43.

    Ramsey FP. Weight or the value of knowledge. The British Journal for the Philosophy of Science. 1990;41:1-4.

    Not in our library10.1093/bjps/41.1.1

44. 44.

    Ravetz JR. Scientific knowledge and its social problems. British Society for the Philosophy of Science. 1971;7:72.

    Not in our library10.1017/s0007087400012875

45. 45.

    Rosenthal R. The file drawer problem and tolerance for null results. Psychological Bulletin. 1979;86:638-641.

    Not in our library10.1037/0033-2909.86.3.638

46. 46.

    Rzhetsky A, Iossifov I, Loh JM, White KP. Microparadigms: Chains of collective reasoning in publications about molecular interactions. PNAS. 2006;103:4940-4945.

    Not in our library10.1073/pnas.0600591103

47. 47.

    Savage LJ. The Foundations of Statistics. 1954.

    Not in our library

48. 48.

    Silvertown J, McConway KJ. Does "publication bias" lead to biased science? Oikos. 1997;79:167-168.

    Not in our library10.2307/3546101

49. 49.

    Simmons LW, Tomkins JL, Kotiaho JS, Hunt J. Fluctuating paradigm. Proceedings of the Royal Society B: Biological Sciences. 1999;266:593-595.

    Not in our library10.1098/rspb.1999.0677

50. 50.

    Smith GD, Ebrahim S. Data dredging, bias, or confounding. BMJ. 2002;325:1437-1438.

    Not in our library10.1136/bmj.325.7378.1437

51. 51.

    Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ. Publication and related biases. Health Technology Assessment. 2000;4:1-115.

    Not in our library

52. 52.

    Sterling TD. Publication decisions and their possible effects on inferences drawn from tests of significance –Or vice versa. Journal of the American Statistical Association. 1959;54:30-34.

    Not in our library10.1080/01621459.1959.10501497

53. 53.

    Tannock IF. False-positive results in clinical trials: Multiple significance tests and the problem of unreported comparisons. Journal of the National Cancer Institute. 1996;88:206-207.

    Not in our library10.1093/jnci/88.3-4.206

54. 54.

    Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine. 2008;358:252-260.

    Not in our library10.1056/nejmsa065779

55. 55.

    van Assen MA, van Aert RC, Nuijten MB, Wicherts JM. Why publishing everything is more effective than selective publishing of statistically significant results. PLoS One. 2014;9:e84896.

    Not in our library10.1371/journal.pone.0084896

Cited by papers we hold 4

• French Bulldogs differ to other dogs in the UK in propensity for many common disorders: a VetCompass study · 2021
  Not checked
• Questionable research practices may have little effect on replicability · 2020
  Not checked
• Open science and modified funding lotteries can impede the natural selection of bad science · 2019
  Not checked
• Ten common statistical mistakes to watch out for when writing or reviewing a manuscript · 2019
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.