There has been discussion in the social and behavioral sciences, the natural and biological sciences, and statistics, about the use of the p-value in guiding decisions based on quantitative research.
In 1996 the American Educational Research Association published an editorial addressing issues with statistical significance. They suggested that there should be an emphasis on effect size, better language around reporting statistical significance, and consider replicability. In 1999 the American Psychological Association published a report outlining ways to increase rigor. One of the recommendations they make is to report effect size and intervals around the effect size. In 2016 the American Statistical Association published a statement on statistical significance and p-values. This work not only clarifies what p-values represent but recommends that potentially researchers should be looking to other measures including confidence intervals and likelihood ratios. They also discuss effect size. In 2019 the ASA followed up with an editorial in which went further, in which they say not to use the term statistical significance:
- “We conclude, based on our review of the articles in this special issue and the broader literature, that it is time to stop using the term “statistically significant” entirely. Nor should variants such as “significantly different,” “p < 0.05,” and “nonsignificant” survive, whether expressed in words, by asterisks in a table, or in some other way (Wassterstein, Shirm, & Lazar, 2019).
Given this information, what should you do? I am not a statistician, I am just a data analyst. At the very least effect sizes should be reported. Much of the literature on this topic refers to effect size. Also, confidence intervals around effect sizes should be reported. Finally, confidence intervals should be reported around test statistics where appropriate.
Here is a website that you can get confidence intervals for effect sizes: Practical Meta-Analysis Effect Size Calculator, and here is a website where you can convert between effect sizes: Psychometrica
Here is a website where you can convert between effect sizes: Psychometrica: https://www.psychometrica.de/effect_size.html
Articles to read:
- American Statistical Association: Moving to a World Beyond “p < 0.05” (2019)
- Unethical numbers? A meta-analysis of library impact studies (2019)
- American Statistical Association: Is it the end of ‘statistical significance’? The battle to make science more uncertain
- Science Magazine: It will be much harder to call new findings ‘signifcant’ if this team gets its way
- American Statistical Association: The ASA Statement on p-Values: Context, Process, and Purpose (2016)
- Currents in Pharmacy Teaching & Learning: Practical significance: Moving beyond statistical significance (2016)
- Science Magazine: Mission Improbable: A Concise and Precise De6nition of P-Value (2009)
- American Psychological Association: Statistical Methods in Psychology Journals
- American Educational Research Association: AERA Editorial Policies regarding Statistical Significance Testing: Three Suggested Reforms (1996)