What’s in a p-value? A fragility analysis of AUA guidelines on BPH

Abstract

Introduction/Objectives: Randomized controlled trials (RCTs) underpin clinical practice guidelines (CPGs) utilized by physicians to direct management of care. Evidence shows that physicians often interpret significant p-values to be “real-world probability” which is not accurate, as p-values can be skewed by confounding variables such as sample size and loss to follow-up. Therefore, there is a need to assess the robustness of endpoints within RCTs that underpin CPGs, specifically for benign prostatic hyperplasia (BPH). This study uses the Fragility Index (FI) and Fragility Quotient (FQ) to assess the strength of statistically significant findings for RCTs cited in the American Urological Association (AUA) guidelines for benign prostatic hyperplasia.

Methods: Two investigators independently screened the AUA guidelines for management of BPH for RCTs cited as evidence for recommendations. In order to be included in this analysis, RCTs needed to 1) report a statistically significant (p ≤ 0.05) dichotomous outcome as a primary or secondary endpoint, 2) have a parallel or two-by-two factorial design, 3) be randomized in approximately a 1:1 ratio, and 4) be available in English. Investigators extracted data related to event rate per group (e.g. incidence of acute urinary retention in each group) and loss to follow-up which was compared against the FI. Stata was used to calculate the FI and FQ which was then summarized and reported according to primary or secondary endpoints.

Results: Among the 373 citations in the AUA guidelines, twenty-four RCTs met inclusion criteria with 29 distinct outcomes analyzed. The median fragility index was 15 (IQR = 4 – 38), indicating that twelve alternative events to either study arm would nullify statistical significance. Six studies had a FI of ≤ 2, indicating that only 1-2 outcomes would need to be changed in order to render non-significance of results. In 10/24 RCTs, the number of patients lost to follow-up was greater than the FI.

Conclusions: The AUA clinical practice guidelines for management of BPH cite RCTs with more robust findings when compared to previous studies assessing fragility in the field of urology. While several included studies had high fragility, the median FI in our analysis was approximately 4-5 times higher than comparable studies of urological RCTs. However, there are areas where improvement is necessary to support the highest quality of evidence-based medicine.