Effect Size & Statistical Significance

There has been discussion in the social and behavioral sciences, the natural and biological sciences, and statistics, about the use of the p-value in guiding decisions based on quantitative research.

In 1996 the American Educational Research Association published an editorial addressing issues with statistical significance. They suggested that there should be an emphasis on effect size, better language around reporting statistical significance, and consider replicability. In 1999 the American Psychological Association published a report outlining ways to increase rigor. One of the recommendations they make is to report effect size and intervals around the effect size. In 2016 the American Statistical Association published a statement on statistical significance and p-values. This work not only clarifies what p-values represent but recommends that potentially researchers should be looking to other measures including confidence intervals and likelihood ratios. They also discuss effect size. In 2019 the ASA followed up with an editorial in which went further, in which they say not to use the term statistical significance:

    “We conclude, based on our review of the articles in this special issue and the broader literature, that it is time to stop using the term “statistically significant” entirely. Nor should variants such as “significantly different,” “p < 0.05,” and “nonsignificant” survive, whether expressed in words, by asterisks in a table, or in some other way (Wassterstein, Shirm, & Lazar, 2019).

Given this information, what should you do? I am not a statistician, I am just a data analyst. At the very least effect sizes should be reported. Much of the literature on this topic refers to effect size. Also, confidence intervals around effect sizes should be reported. Finally, confidence intervals should be reported around test statistics where appropriate.

Here is a website that you can get confidence intervals for effect sizes: Practical Meta-Analysis Effect Size Calculator, and here is a website where you can convert between effect sizes: Psychometrica

Here is a website where you can convert between effect sizes: Psychometrica: https://www.psychometrica.de/effect_size.html

Articles to read: