No retraction, correction or concern found when last checked on Oct 24, 2026.

Semantic and cognitive tools to aid statistical science: replace confidence and significance by compatibility and surprise

Zad Rafi, Sander Greenland

BMC Medical Research Methodology · 2020 · Review · CC BY

Abstract

Background. Researchers often misinterpret and misrepresent statistical outputs. This abuse has led to a large literature on modification or replacement of testing thresholds and P-values with confidence intervals, Bayes factors, and other devices. Because the core problems appear cognitive rather than statistical, we review some simple methods to aid researchers in interpreting statistical outputs. These methods emphasize logical and information concepts over probability, and thus may be more robust to common misinterpretations than are traditional descriptions.

Methods. We use the Shannon transform of the P-value p, also known as the binary surprisal or S-value s = −log2(p), to provide a measure of the information supplied by the testing procedure, and to help calibrate intuitions against simple physical experiments like coin tossing. We also use tables or graphs of test statistics for alternative hypotheses, and interval estimates for different percentile levels, to thwart fallacies arising from arbitrary dichotomies. Finally, we reinterpret P-values and interval estimates in unconditional terms, which describe compatibility of data with the entire set of analysis assumptions. We illustrate these methods with a reanalysis of data from an existing record-based cohort study.

Conclusions. In line with other recent recommendations, we advise that teaching materials and research reports discuss P-values as measures of compatibility rather than significance, compute P-values for alternative hypotheses whenever they are computed for null hypotheses, and interpret interval estimates as showing values of high compatibility with data, rather than regions of confidence. Our recommendations emphasize cognitive devices for displaying the compatibility of the observed data with various hypotheses of interest, rather than focusing on single hypothesis tests or interval estimates. We believe these simple reforms are well worth the minor effort they require.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 91

1.
Greenland S. Invited commentary: the need for cognitive science in methodology. Am J Epidemiol. 2017;186:639-645.
Not in our library10.1093/aje/kwx259
2.
Gigerenzer G. Mindless statistics. J Socio-Econ. 2004;33:587-606.
Not in our library10.1016/j.socec.2004.09.033
3.
Stark PB, Saltelli A. Cargo-cult statistics and scientific crisis. Significance. 2018;15:40-43.
Not in our library10.1111/j.1740-9713.2018.01174.x
4.
Simmons JP, Nelson LD, Simonsohn U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci. 2011;22:1359-1366.
Not in our library10.1177/0956797611417632
5.
Estimating the reproducibility of psychological science · 2015
No notice
6.
The Economics of Reproducibility in Preclinical Research · 2015
Corrected
7.
Camerer CF, Dreber A, Forsell E, Ho T-H, Huber J, Johannesson M, et al. Evaluating replicability of laboratory experiments in economics. Science. 2016;351:1433-1436.
Not in our library10.1126/science.aaf0918
8.
Lash TL, Collin LJ, Van Dyke ME. The replication crisis in epidemiology: snowball, snow job, or winter solstice? Curr Epidemiol Rep. 2018;5:175-183.
Not in our library10.1007/s40471-018-0148-x
9.
Cassidy SA, Dimova R, Giguère B, Spence JR, Stanley DJ. Failing grade: 89% of introduction-to-psychology textbooks that define or explain statistical significance do so incorrectly. Adv Methods Pract Psychol Sci. 2019. 10.1177/2515245919858072.
Not in our library
10.
Leek JT, Peng RD. Statistics: P values are just the tip of the iceberg. Nat News. 2015;520:612.
Not in our library10.1038/520612a
11.
Lang JM, Rothman KJ, Cann CI. That confounded P-value. Epidemiology. 1998;9:7-8.
Not in our library10.1097/00001648-199801000-00004
12.
Pearson KV. Note on the significant or non-significant character of a sub-sample drawn from a sample. Biometrika. 1906;5:181-183.
Not in our library
13.
Boring EG. Mathematical vs. scientific significance. Psychol Bull. 1919;16:335-338.
Not in our library10.1037/h0074554
14.
Tyler RW. What is statistical significance? Educ Res Bull. 1931;10:115-142.
Not in our library
15.
Benjamin DJ, Berger JO, Johannesson M, Nosek BA, Wagenmakers E, Berk R, et al. Redefine statistical significance. Nat Hum Behav. 2017;2:6-10.
Not in our library10.1038/s41562-017-0189-z
16.
Lakens D, Adolfi FG, Albers CJ, Anvari F, Apps MAJ, Argamon SE, et al. Justify your alpha. Nat Hum Behav. 2018;2:168-171.
Not in our library10.1038/s41562-018-0311-x
17.
Lakens D, Scheel AM, Isager PM. Equivalence testing for psychological research: a tutorial. Adv Methods Pract Psychol Sci. 2018;1:259-269.
Not in our library10.1177/2515245918770963
18.
Mayo DG. Statistical inference as severe testing: how to get beyond the statistics wars: Cambridge University Press; 2018. 10.1017/9781107286184.
Not in our library
19.
Rothman KJ. A show of confidence. N Engl J Med. 1978;299:1362-1363.
Not in our library10.1056/nejm197812142992410
20.
Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res. 1999;8:135-160.
Not in our library10.1177/096228029900800204
21.
Cumming G. Understanding the new statistics: effect sizes, confidence intervals, and meta-analysis. Routledge. 2012. 10.4324/9780203807002.
Not in our library
22.
Colquhoun D. The false positive risk: a proposal concerning what to do about p-values. Am Stat. 2019;73:192-201.
Not in our library10.1080/00031305.2018.1529622
23.
Goodman SN. Introduction to Bayesian methods I: measuring the strength of evidence. Clin Trials. 2005;2. 10.1191/1740774505cn098oa.
Not in our library10.1191/1740774505cn098oa
24.
Held L. A new standard for the analysis and design of replication studies. J R Stat Soc Ser A Stat Soc. 2020;183:431-448.
Not in our library10.1111/rssa.12493
25.
Matthews RAJ. Moving towards the post p < 0.05 era via the analysis of credibility. Am Stat. 2019;73:202-212.
Not in our library10.1080/00031305.2018.1543136
26.
Sellke T, Bayarri MJ, Berger JO. Calibration of ρ values for testing precise null hypotheses. Am Stat. 2001;55:62-71.
Not in our library10.1198/000313001300339950
27.
Wang MQ, Yan AF, Katz RV. Researcher requests for inappropriate analysis and reporting: A U.S. survey of consulting biostatisticians. Ann Intern Med. 2018;169:554.
Not in our library10.7326/m18-1230
28.
Gelman A. The problems with P-values are not just with P-values. Am Stat. 2016;70 https://stat.columbia.edu/~gelman/research/published/asa_pvalues.pdf.
Not in our library
29.
Amrhein V, Greenland S, McShane B. Scientists rise up against statistical significance. Nature. 2019;567:305.
Not in our library10.1038/d41586-019-00857-9
30.
Greenland S. Are confidence intervals better termed “uncertainty intervals”? No: call them compatibility intervals. BMJ. 2019;366. 10.1136/bmj.l5381.
Not in our library10.1136/bmj.l5381
31.
Greenland S. Valid P-values behave exactly as they should: some misleading criticisms of P-values and their resolution with S-values. Am Stat. 2019;73:106-114.
Not in our library10.1080/00031305.2018.1529625
32.
Cole SR, Edwards JK, Greenland S. Surprise! Am J Epidemiol. 2020; doi: https://doi.org/10/gg63md.
Not in our library10.1093/aje/kwaa136
33.
Greenland S, Rafi Z. To aid scientific inference, emphasize unconditional descriptions of statistics. ArXiv190908583 StatME. 2020; https://arxiv.org/abs/1909.08583.
Not in our library
34.
Brown HK, Ray JG, Wilton AS, Lunsky Y, Gomes T, Vigod SN. Association between serotonergic antidepressant use during pregnancy and autism spectrum disorder in children. J Am Med Assoc. 2017;317:1544-1552.
Not in our library10.1001/jama.2017.3415
35.
Brown HK, Hussain-Shamsy N, Lunsky Y, Dennis C-LE, Vigod SN. The association between antenatal exposure to selective serotonin reuptake inhibitors and autism: a systematic review and meta-analysis. J Clin Psychiatry. 2017;78:e48-e58.
Not in our library10.4088/jcp.15r10194
36.
Yasgur B. Antidepressants in pregnancy: no link to autism. 2017.
Not in our library
37.
Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ. 1995;311:485.
Not in our library10.1136/bmj.311.7003.485
38.
The earth is flat ( p > 0.05): significance thresholds and the crisis of unreplicable research · 2017
No notice
39.
McShane BB, Gal D, Gelman A, Robert C, Tackett JL. Abandon Statistical Significance. Am Stat. 2019;73:235-245.
Not in our library10.1080/00031305.2018.1527253
40.
Poole C. Beyond the confidence interval. Am J Public Health. 1987;77:195-199.
Not in our library10.2105/ajph.77.2.195

All 91 references

Cited by papers we hold 1

Replacing statistical significance and non-significance with better approaches to sampling uncertainty · 2022
Not checked

Reads

Read by people on this page

None yet.

Checked or read by AI through naulon

None yet.

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.

Are you an author?

Is this your paper?

Claim it to add your note and be paid when agents cite it. Free.