There has been discussion in the social and behavioral sciences, the natural and biological sciences, and statistics, about the use of the p-value in guiding decisions based on quantitative research.
In 1996 the American Educational Research Association (AERA) published an editorial addressing issues with statistical significance. They suggested that there should be an emphasis on effect size, better language around reporting statistical significance, and consideration of replicability. In 1999 the American Psychological Association (APA) published a report outlining ways to increase rigor. One of the recommendations they make is to report effect size and confidence intervals around the effect size. In 2016 the American Statistical Association (ASA) published a statement on statistical significance and p-values. This work not only clarifies what p-values represent but recommends that researchers should be looking to other measures including confidence intervals and likelihood ratios; they also discuss effect size. In 2019 the ASA followed up with an editorial in which went further, in which they say not to use the term statistical significance:
- “We conclude, based on our review of the articles in this special issue and the broader literature, that it is time to stop using the term “statistically significant” entirely. Nor should variants such as “significantly different,” “p < 0.05,” and “nonsignificant” survive, whether expressed in words, by asterisks in a table, or in some other way (Wassterstein, Shirm, & Lazar, 2019).
Given this information, what should you do? I am not a statistician, I am just a data analyst. At the very least effect sizes should be reported. Also, confidence intervals for effect sizes should be reported. Finally, confidence intervals should be reported for test statistics where appropriate.
Here is a website that you can get confidence intervals for effect sizes: Practical Meta-Analysis Effect Size Calculator,
Here is a website where you can convert between effect sizes: Psychometrica
Articles to read:
- American Statistical Association: Moving to a World Beyond “p < 0.05” (2019)
- Unethical numbers? A meta-analysis of library impact studies (2019)
- American Statistical Association: Is it the end of ‘statistical significance’? The battle to make science more uncertain
- Science Magazine: It will be much harder to call new findings ‘signifcant’ if this team gets its way
- American Statistical Association: The ASA Statement on p-Values: Context, Process, and Purpose (2016)
- Currents in Pharmacy Teaching & Learning: Practical significance: Moving beyond statistical significance (2016)
- Science Magazine: Mission Improbable: A Concise and Precise De6nition of P-Value (2009)
- American Psychological Association: Statistical Methods in Psychology Journals
- American Educational Research Association: AERA Editorial Policies regarding Statistical Significance Testing: Three Suggested Reforms (1996)