What is it about?
The article argues that the common definition of statistical significance, based solely on a p-value of 0.05 or less, is inadequate. The authors explain that statistical significance also depends on the power of the study. Power refers to the chance that a study will detect a true effect if it exists. The authors propose a new way to define statistical significance that accounts for both p-values and study power. Their definition is based on the positive predictive value, which is the probability that a statistically significant finding reflects a true effect rather than chance. According to their definition, the power of the study and the p-value must be considered together to determine if a result is truly statistically significant. As study power decreases, the p-value would need to be lower to achieve statistical significance. For example, with 80% power, a p-value of 0.042 or less may be needed rather than the conventional 0.05. The authors argue this new definition provides a more complete and clinically useful understanding of statistical significance by explicitly incorporating study power. Their approach aims to avoid conclusions based on weak studies with low statistical power.
Featured Image
Photo by Edge2Edge Media on Unsplash
Why is it important?
- It highlights flaws in the conventional definition of statistical significance (p<0.05), which does not account for study power. This can lead to improper interpretation of research findings. - The proposed new definition provides a more nuanced, accurate understanding of statistical significance by incorporating both p-values and power. This could improve the validity of conclusions drawn from research. - Considering study power is crucial to avoid false positive or negative results, especially in studies with small sample sizes or minimal effects. The new definition formally builds in this important factor. - Inadequate power is a common problem in research, so using a definition of statistical significance that accounts for power could raise standards and improve reproducibility. - The new approach links statistical significance more directly to clinical relevance, reducing the gap between statistical findings and practice. - By adjusting p-value thresholds based on power, the definition requires stronger evidence from weaker studies and helps avoid overinterpreting underpowered results. - The ideas could influence future guidelines and practice around statistical analysis, study design, and evidence standards in research. In summary, the proposed definition better captures the true meaning of statistical significance and has implications for improving research quality and clinical applicability. Formalizing the role of power addresses a major limitation of current methods.
Perspectives
I believe our new definition of statistical significance represents an important advancement in the field of biostatistics. After years of seeing questionable conclusions drawn from underpowered studies, I felt a responsibility to help address this issue. Our goal with this work was to move statistical significance more closely with true clinical meaningfulness. The standard of p<0.05 alone is outdated and inadequate. My co-author and I strove to incorporate study power into the significance definition formally and quantitatively. Developing the mathematical basis for this new definition took time and care. But it's a concept I firmly believe in - we need to raise the bar for declaring results statistically significant, not lower it. This will require careful attention to power analysis during study design. Some statisticians may resist altering such an entrenched idea as the p-value cutoff. But I hope this paper further examines how we can improve research practices. The p-value alone is a dangerous basis for conclusions. I believe that with thoughtful adjustments like those we propose, great progress can be made toward more reproducible, meaningful science. On a personal level, questioning established norms is difficult. Yet the shortcomings of the old definitions are troublesome. We struck a constructive tone in this paper. The goal is to propel statistics forward. With continued open-minded debate, I'm confident better understanding will prevail. There are always improvements to be made, even to our proposed methods. I'm excited to see how these conversations unfold.
Thomas F Heston MD
University of Washington
Read the Original
This page is a summary of: Predictive power of statistical significance, World Journal of Methodology, December 2017, Baishideng Publishing Group Co., Limited (formerly WJG Press),
DOI: 10.5662/wjm.v7.i4.112.
You can read the full text:
Resources
Contributors
The following have contributed to this page
