Statistical significance and sports medicine trials

Abstract

Objectives: Lowering the threshold for statistical significance in medical research from a P value of .05 to .005 was recently proposed to reduce misinterpretation of study results. What effect this proposal would have on orthopaedic sports medicine literature is currently unclear.

Research Question/Hypothesis: We seek to determine how the newly proposed threshold could affect the interpretation of previously published sports medicine RCTs.

Methods: We searched PubMed from January 01, 2016 to December 31, 2017 for RCTs published in the American Journal of Sports Medicine, Arthroscopy, and Knee Surgery, Sports Traumatology, Arthroscopy. We extracted P value data for primary endpoints, since RCTs are most often powered for these endpoints. We used Google Forms for data extraction and STATA 13.1 for the data analysis.

Results: Of the 159 studies, only 13 (8%) of the studies have endpoints in which all P values are below the new threshold of .005. 40 (25%) of the studies have endpoints in which some would meet the new P value threshold of .005, and some would not meet this new threshold. 106 (67%) of the studies have no endpoints in which the P value(s) was less than .005. Overall, 38% (59/157) of the previously statistically significant primary endpoints were less than .005, while 62% (98/157) would be reclassified as suggestive.

Conclusions: Of statistically significant endpoints in our sample, only 17% (59/350) would maintain their statistical significance with a P value threshold of less than .005, and only 8% of studies would maintain their overall significance with all P values falling below the new threshold.