No retraction, correction or concern found when last checked on Oct 24, 2026.

Semantic and cognitive tools to aid statistical science: replace confidence and significance by compatibility and surprise

Zad Rafi, Sander Greenland

BMC Medical Research Methodology · 2020 · Review · CC BY

Abstract

Background. Researchers often misinterpret and misrepresent statistical outputs. This abuse has led to a large literature on modification or replacement of testing thresholds and P-values with confidence intervals, Bayes factors, and other devices. Because the core problems appear cognitive rather than statistical, we review some simple methods to aid researchers in interpreting statistical outputs. These methods emphasize logical and information concepts over probability, and thus may be more robust to common misinterpretations than are traditional descriptions.

Methods. We use the Shannon transform of the P-value p, also known as the binary surprisal or S-value s = −log2(p), to provide a measure of the information supplied by the testing procedure, and to help calibrate intuitions against simple physical experiments like coin tossing. We also use tables or graphs of test statistics for alternative hypotheses, and interval estimates for different percentile levels, to thwart fallacies arising from arbitrary dichotomies. Finally, we reinterpret P-values and interval estimates in unconditional terms, which describe compatibility of data with the entire set of analysis assumptions. We illustrate these methods with a reanalysis of data from an existing record-based cohort study.

Conclusions. In line with other recent recommendations, we advise that teaching materials and research reports discuss P-values as measures of compatibility rather than significance, compute P-values for alternative hypotheses whenever they are computed for null hypotheses, and interpret interval estimates as showing values of high compatibility with data, rather than regions of confidence. Our recommendations emphasize cognitive devices for displaying the compatibility of the observed data with various hypotheses of interest, rather than focusing on single hypothesis tests or interval estimates. We believe these simple reforms are well worth the minor effort they require.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 91

1.
Greenland S. Invited commentary: the need for cognitive science in methodology. Am J Epidemiol. 2017;186:639-645.
Not in our library10.1093/aje/kwx259
2.
Gigerenzer G. Mindless statistics. J Socio-Econ. 2004;33:587-606.
Not in our library10.1016/j.socec.2004.09.033
3.
Stark PB, Saltelli A. Cargo-cult statistics and scientific crisis. Significance. 2018;15:40-43.
Not in our library10.1111/j.1740-9713.2018.01174.x
4.
Simmons JP, Nelson LD, Simonsohn U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci. 2011;22:1359-1366.
Not in our library10.1177/0956797611417632
5.
Estimating the reproducibility of psychological science · 2015
No notice
6.
The Economics of Reproducibility in Preclinical Research · 2015
Corrected
7.
Camerer CF, Dreber A, Forsell E, Ho T-H, Huber J, Johannesson M, et al. Evaluating replicability of laboratory experiments in economics. Science. 2016;351:1433-1436.
Not in our library10.1126/science.aaf0918
8.
Lash TL, Collin LJ, Van Dyke ME. The replication crisis in epidemiology: snowball, snow job, or winter solstice? Curr Epidemiol Rep. 2018;5:175-183.
Not in our library10.1007/s40471-018-0148-x
9.
Cassidy SA, Dimova R, Giguère B, Spence JR, Stanley DJ. Failing grade: 89% of introduction-to-psychology textbooks that define or explain statistical significance do so incorrectly. Adv Methods Pract Psychol Sci. 2019. 10.1177/2515245919858072.
Not in our library
10.
Leek JT, Peng RD. Statistics: P values are just the tip of the iceberg. Nat News. 2015;520:612.
Not in our library10.1038/520612a
11.
Lang JM, Rothman KJ, Cann CI. That confounded P-value. Epidemiology. 1998;9:7-8.
Not in our library10.1097/00001648-199801000-00004
12.
Pearson KV. Note on the significant or non-significant character of a sub-sample drawn from a sample. Biometrika. 1906;5:181-183.
Not in our library
13.
Boring EG. Mathematical vs. scientific significance. Psychol Bull. 1919;16:335-338.
Not in our library10.1037/h0074554
14.
Tyler RW. What is statistical significance? Educ Res Bull. 1931;10:115-142.
Not in our library
15.
Benjamin DJ, Berger JO, Johannesson M, Nosek BA, Wagenmakers E, Berk R, et al. Redefine statistical significance. Nat Hum Behav. 2017;2:6-10.
Not in our library10.1038/s41562-017-0189-z
16.
Lakens D, Adolfi FG, Albers CJ, Anvari F, Apps MAJ, Argamon SE, et al. Justify your alpha. Nat Hum Behav. 2018;2:168-171.
Not in our library10.1038/s41562-018-0311-x
17.
Lakens D, Scheel AM, Isager PM. Equivalence testing for psychological research: a tutorial. Adv Methods Pract Psychol Sci. 2018;1:259-269.
Not in our library10.1177/2515245918770963
18.
Mayo DG. Statistical inference as severe testing: how to get beyond the statistics wars: Cambridge University Press; 2018. 10.1017/9781107286184.
Not in our library
19.
Rothman KJ. A show of confidence. N Engl J Med. 1978;299:1362-1363.
Not in our library10.1056/nejm197812142992410
20.
Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res. 1999;8:135-160.
Not in our library10.1177/096228029900800204
21.
Cumming G. Understanding the new statistics: effect sizes, confidence intervals, and meta-analysis. Routledge. 2012. 10.4324/9780203807002.
Not in our library
22.
Colquhoun D. The false positive risk: a proposal concerning what to do about p-values. Am Stat. 2019;73:192-201.
Not in our library10.1080/00031305.2018.1529622
23.
Goodman SN. Introduction to Bayesian methods I: measuring the strength of evidence. Clin Trials. 2005;2. 10.1191/1740774505cn098oa.
Not in our library10.1191/1740774505cn098oa
24.
Held L. A new standard for the analysis and design of replication studies. J R Stat Soc Ser A Stat Soc. 2020;183:431-448.
Not in our library10.1111/rssa.12493
25.
Matthews RAJ. Moving towards the post p < 0.05 era via the analysis of credibility. Am Stat. 2019;73:202-212.
Not in our library10.1080/00031305.2018.1543136
26.
Sellke T, Bayarri MJ, Berger JO. Calibration of ρ values for testing precise null hypotheses. Am Stat. 2001;55:62-71.
Not in our library10.1198/000313001300339950
27.
Wang MQ, Yan AF, Katz RV. Researcher requests for inappropriate analysis and reporting: A U.S. survey of consulting biostatisticians. Ann Intern Med. 2018;169:554.
Not in our library10.7326/m18-1230
28.
Gelman A. The problems with P-values are not just with P-values. Am Stat. 2016;70 https://stat.columbia.edu/~gelman/research/published/asa_pvalues.pdf.
Not in our library
29.
Amrhein V, Greenland S, McShane B. Scientists rise up against statistical significance. Nature. 2019;567:305.
Not in our library10.1038/d41586-019-00857-9
30.
Greenland S. Are confidence intervals better termed “uncertainty intervals”? No: call them compatibility intervals. BMJ. 2019;366. 10.1136/bmj.l5381.
Not in our library10.1136/bmj.l5381
31.
Greenland S. Valid P-values behave exactly as they should: some misleading criticisms of P-values and their resolution with S-values. Am Stat. 2019;73:106-114.
Not in our library10.1080/00031305.2018.1529625
32.
Cole SR, Edwards JK, Greenland S. Surprise! Am J Epidemiol. 2020; doi: https://doi.org/10/gg63md.
Not in our library10.1093/aje/kwaa136
33.
Greenland S, Rafi Z. To aid scientific inference, emphasize unconditional descriptions of statistics. ArXiv190908583 StatME. 2020; https://arxiv.org/abs/1909.08583.
Not in our library
34.
Brown HK, Ray JG, Wilton AS, Lunsky Y, Gomes T, Vigod SN. Association between serotonergic antidepressant use during pregnancy and autism spectrum disorder in children. J Am Med Assoc. 2017;317:1544-1552.
Not in our library10.1001/jama.2017.3415
35.
Brown HK, Hussain-Shamsy N, Lunsky Y, Dennis C-LE, Vigod SN. The association between antenatal exposure to selective serotonin reuptake inhibitors and autism: a systematic review and meta-analysis. J Clin Psychiatry. 2017;78:e48-e58.
Not in our library10.4088/jcp.15r10194
36.
Yasgur B. Antidepressants in pregnancy: no link to autism. 2017.
Not in our library
37.
Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ. 1995;311:485.
Not in our library10.1136/bmj.311.7003.485
38.
The earth is flat ( p > 0.05): significance thresholds and the crisis of unreplicable research · 2017
No notice
39.
McShane BB, Gal D, Gelman A, Robert C, Tackett JL. Abandon Statistical Significance. Am Stat. 2019;73:235-245.
Not in our library10.1080/00031305.2018.1527253
40.
Poole C. Beyond the confidence interval. Am J Public Health. 1987;77:195-199.
Not in our library10.2105/ajph.77.2.195
41.
Rothman KJ. Significance Questing. Ann Intern Med. 1986;105:445-447.
Not in our library10.7326/0003-4819-105-3-445
42.
Wasserstein RL, Schirm AL, Lazar NA. Moving to a world beyond “p < 0.05”. Am Stat. 2019;73:1-19.
Not in our library10.1080/00031305.2019.1583913
43.
Fisher RA. Statistical methods for research workers. 1925.
Not in our library
44.
Pearson KX. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Lond Edinb Dublin Philos Mag J Sci. 1900;50:157-175.
Not in our library10.1080/14786440009463897
45.
Stigler SM. Attempts to Revive the Binomial. In: The history of statistics: the measurement of uncertainty before 1900: Harvard University Press; 1986. https://books.google.com/books?id=M7yvkERHIIMC.
Not in our library
46.
Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations · 2016
No notice
47.
P-values as percentiles · 2015
Not checked
48.
Vos P, Holbert D. Frequentist inference without repeated sampling. ArXiv190608360 StatOT. 2019; https://arxiv.org/abs/1906.08360.
Not in our library
49.
Shannon CE. A mathematical theory of communication. Bell Syst Tech J. 1948;27:379-423.
Not in our library10.1002/j.1538-7305.1948.tb01338.x
50.
Good IJ. The surprise index for the multivariate normal distribution. Ann Math Stat. 1956;27:1130–5. Corrigendum Ann Math Stat. 1957;28:1055.
Not in our library
51.
Cousins RD. The Jeffreys–Lindley paradox and discovery criteria in high energy physics. Synthese. 2017;194:395-432.
Not in our library10.1007/s11229-014-0525-z
52.
Estimation of significance thresholds for genomewide association scans · 2008
No notice
53.
Hand DJ. The improbability principle: why coincidences, miracles, and rare events happen every day: Macmillan; 2014. https://books.google.com/books?id=raZRAQAAQBAJ.
Not in our library
54.
Bowley AL. Discussion on Dr. Neyman’s Paper. P. 607–610 in: Neyman J. On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection (with discussion). J R Stat Soc. 1934;4:558-625.
Not in our library
55.
Cox DR, Hinkley DV. Chapter 7, interval estimation. In: Theoretical Statistics: Chapman and Hall/CRC; 1974. p. 207–49. 10.1201/b14832.
Not in our library
56.
Cox DR. Principles of statistical inference: Cambridge University Press; 2006. 10.1017/cbo9780511813559.
Not in our library
57.
Amrhein V, Trafimow D, Greenland S. Inferential statistics as descriptive statistics: there is no replication crisis if we don’t expect replication. Am Stat. 2019;73:262-270.
Not in our library10.1080/00031305.2018.1543137
58.
Poole C. Confidence intervals exclude nothing. Am J Public Health. 1987;77:492-493.
Not in our library10.2105/ajph.77.4.492
59.
Gelman A, Stern H. The difference between “significant” and “not significant” is not itself statistically significant. Am Stat. 2006;60:328-331.
Not in our library10.1198/000313006x152649
60.
Birnbaum A. A unified theory of estimation. I Ann Math Stat. 1961;32:112-135.
Not in our library10.1214/aoms/1177705145
61.
Sullivan KM, Foster DA. Use of the confidence interval function. Epidemiology. 1990;1:39-42.
Not in our library10.1097/00001648-199001000-00009
62.
Rothman KJ, Greenland S, Lash TL. Precision and statistics in epidemiologic studies. In: Rothman KJ, Greenland S, Lash TL, editors. Modern Epidemiology. 3rd edition: Lippincott Williams & Wilkins; 2008. p. 148–67. https://books.google.com/books/about/Modern_Epidemiology.html?id=Z3vjT9ALxHUC.
Not in our library
63.
Rafi Z, Vigotsky AD. concurve: Computes and Plots Consonance (Confidence) Intervals, P-Values, and S-Values to Form Consonance and Surprisal Functions. R. CRAN. 2020.
Not in our library
64.
Rücker G, Schwarzer G. Beyond the forest plot: the drapery plot. Res Synth Methods. 2020. 10.1002/jrsm.1410.
Not in our library10.1002/jrsm.1410
65.
Fraser DAS. The P-value function and statistical inference. Am Stat. 2019;73:135-147.
Not in our library10.1080/00031305.2018.1556735
66.
Whitehead J. The case for frequentism in clinical trials. Stat Med. 1993;12:1405-1413.
Not in our library10.1002/sim.4780121506
67.
Xie M, Singh K. Confidence distribution, the Frequentist distribution estimator of a parameter: a review. Int Stat Rev. 2013;81:3-39.
Not in our library10.1111/insr.12000
68.
Singh K, Xie M, Strawderman WE. Confidence distribution (CD) – distribution estimator of a parameter. 2007.
Not in our library
69.
Schweder T, Hjort NL. Confidence, likelihood, probability: statistical inference with confidence distributions: Cambridge University Press; 2016. https://books.google.com/books/about/Confidence_Likelihood_Probability.html?id=t7KzCwAAQBAJ.
Not in our library
70.
Rubenstein S. A new low in drug research: 21 fabricated studies. WSJ. 2009; https://blogs.wsj.com/health/2009/03/11/a-new-low-in-drug-research-21-fabricated-studies/.
Not in our library
71.
Schmidt M, Rothman KJ. Mistaken inference caused by reliance on and misinterpretation of a significance test. Int J Cardiol. 2014;177:1089-1090.
Not in our library10.1016/j.ijcard.2014.09.205
72.
Greenland S. A serious misinterpretation of a consistent inverse association of statin use with glioma across 3 case-control studies. Eur J Epidemiol. 2017;32:87-88.
Not in our library10.1007/s10654-016-0205-z
73.
Bauchner H, Golub RM, Fontanarosa PB. Reporting and interpretation of randomized clinical trials. J Am Med Assoc. 2019;322:732-735.
Not in our library10.1001/jama.2019.12056
74.
Tversky A, Kahneman D. Judgment under uncertainty: heuristics and biases. Science. 1974;185:1124-1131.
Not in our library10.1126/science.185.4157.1124
75.
Gilovich T, Griffin D, Kahneman D. Heuristics and biases: the psychology of intuitive judgment: Cambridge University Press; 2002. https://books.google.com/books/about/Heuristics_and_Biases.html?id=FfTVDY-zrCoC.
Not in our library
76.
Gigerenzer G, Marewski JN. Surrogate science: the idol of a universal method for scientific inference. J Manag. 2015;41:421-440.
Not in our library
77.
Greenland S, Poole C. Living with p values: resurrecting a Bayesian perspective on frequentist statistics. Epidemiology. 2013;24:62-68.
Not in our library10.1097/ede.0b013e3182785741
78.
Greenland S, Rothman KJ. Fundamentals of epidemiologic data analysis. In: Rothman KJ, Greenland S, Lash TL, editors. Modern Epidemiology. 3rd edition: Lippincott Williams & Wilkins; 2008. p. 213–37. https://books.google.com/books/about/Modern_Epidemiology.html?id=Z3vjT9ALxHUC.
Not in our library
79.
Cox DR, Hinkley DV. Chapter 3, pure significance tests. In: Theoretical Statistics: Chapman and Hall/CRC; 1974. p. 64–87. 10.1201/b14832.
Not in our library
80.
Bayarri MJ, Berger JO. P values for composite null models. J Am Stat Assoc. 2000;95:1127-1142.
Not in our library
81.
Robins JM, van der Vaart A, Ventura V. Asymptotic distribution of P values in composite null models. J Am Stat Assoc. 2000;95:1143-1156.
Not in our library
82.
Stark PB. Constraints versus priors. SIAMASA J Uncertain Quantif. 2015;3:586-598.
Not in our library10.1137/130920721
83.
Cox DR. A note on partially Bayes inference and the linear model. Biometrika. 1975;62:651-654.
Not in our library10.1093/biomet/62.3.651
84.
Greenland S. Randomization, statistics, and causal inference. Epidemiology. 1990;1:421-429.
Not in our library10.1097/00001648-199011000-00003
85.
Greenland S, Robins J. Identifiability, exchangeability, and epidemiological confounding. Int J Epidemiol. 1986;15:413-419.
Not in our library10.1093/ije/15.3.413
86.
Gelman A, Carlin J. Beyond power calculations: assessing type S (sign) and type M (magnitude) errors. Perspect Psychol Sci. 2014;9:641-651.
Not in our library10.1177/1745691614551642
87.
Westreich D, Greenland S. The table 2 fallacy: presenting and interpreting confounder and modifier coefficients. Am J Epidemiol. 2013;177:292-298.
Not in our library10.1093/aje/kws412
88.
Greenland S, Hofman A. Multiple comparisons controversies are about context and costs, not frequentism versus Bayesianism. Eur J Epidemiol. 2019. 10.1007/s10654-019-00552-z.
Not in our library10.1007/s10654-019-00552-z
89.
Greenland S. Analysis goals, error-cost sensitivity, and analysis hacking: essential considerations in hypothesis testing and multiple comparisons. Ped Perinatal Epidemiol. 2020; in press.
Not in our library10.1111/ppe.12711
90.
McShane BB, Gal D. Statistical significance and the dichotomization of evidence. J Am Stat Assoc. 2017;112:885-895.
Not in our library10.1080/01621459.2017.1289846
91.
Folks L. Ideas of statistics: Wiley; 1981. https://books.google.com/books/about/Ideas_of_statistics.html?id=Bn8pAQAAMAAJ.
Not in our library

Cited by papers we hold 1

Replacing statistical significance and non-significance with better approaches to sampling uncertainty · 2022
Not checked

Reads

Read by people on this page

None yet.

Checked or read by AI through naulon

None yet.

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.

Are you an author?

Is this your paper?

Claim it to add your note and be paid when agents cite it. Free.