www.cremieux.xyz/p/statins-and-strength?utm_source=substack&utm_medium=email
3 corrections found
significance has to do with the p-value being below a given threshold, not at it.
That statement is not generally correct: in standard hypothesis testing, a p-value equal to the prespecified alpha is conventionally treated as meeting the significance threshold (p ≤ α), not failing it.
Full reasoning
The article makes a categorical statistical claim here that is wrong on its own terms.
In standard null-hypothesis significance testing, the usual decision rule is reject when p ≤ α. Authoritative references explicitly describe the threshold this way, not as requiring a p-value to be strictly below alpha.
That matters especially here because the STOMP paper the article is discussing states in its methods that "statistical significance [was] set at alpha = 0.05" and then reports the myalgia comparison as "p = 0.05". So even if one wants to argue that the unrounded p-value may have been slightly above 0.05, the blanket statement that significance must be below the threshold and not at it is false.
In other words, there are two separate issues:
- Whether the exact, unrounded p-value was actually above 0.05 for that comparison.
- Whether a p-value equal to the prespecified threshold counts as significant.
The quoted sentence gets issue #2 wrong. Under the conventional rule, a p-value equal to alpha is treated as meeting the threshold.
2 sources
- Reproducibility and Replicability in Science (National Academies of Sciences, Engineering, and Medicine)
"The threshold for statistical significance is often referred to as p 'less than' 0.05; we refer to this threshold as 'less than or equal to.'"
- The Effect of Statins on Skeletal Muscle Function
The paper states: "All tests were two-sided with statistical significance set at alpha=0.05." It also reports: "Nineteen atorvastatin and 10 placebo subjects ultimately met the study definition for myalgia (Χ2 = 3.74; p = 0.05)."
not how the authors reported it
This misstates the paper. The STOMP authors did report the myalgia result as significant, giving p = 0.05 and concluding that the trial confirmed increased muscle complaints with atorvastatin.
Full reasoning
This phrase is contradicted by the study the article is discussing.
In the STOMP paper, the authors explicitly wrote that more atorvastatin than placebo subjects developed myalgia (19 vs 10; p = 0.05). They also stated in the methods that statistical significance was set at alpha = 0.05. In the abstract conclusion, they further wrote that the trial "confirms" that statins increase muscle complaints.
So whatever one thinks about the exact unrounded p-value, continuity corrections, or alternative tests, it is simply incorrect to say that a non-significant result was "not how the authors reported it." They reported the comparison as p = 0.05 and interpreted the study as confirming increased muscle complaints on atorvastatin.
1 source
- The Effect of Statins on Skeletal Muscle Function
Abstract: "There were no significant changes in several measures of muscle strength or exercise capacity with atorvastatin, but more atorvastatin than placebo subjects developed myalgia (19 vs 10; p = 0.05)." Methods: "All tests were two-sided with statistical significance set at alpha=0.05." Conclusions: "this blinded, controlled trial confirms the undocumented impression that statins increase muscle complaints."
in which 420 participants who had never taken statins were randomized to either a placebo or high-dose (80mg) daily atorvastatin for six months.
The STOMP paper reports that 468 subjects were randomized, not 420. The 420 figure is not the randomized sample size.
Full reasoning
This sentence misstates the trial's enrollment/randomization numbers.
In the STOMP paper's methods, the authors say: "A total of 468 subjects were randomized" to atorvastatin or placebo. They then note that 203 atorvastatin and 214 placebo participants provided complete data for analysis. So 420 was not the number randomized.
The article's summary therefore conflates a smaller figure mentioned elsewhere with the study's actual randomization count. If the claim is specifically about how many participants were randomized, the correct number from the paper is 468, not 420.
1 source
- The Effect of Statins on Skeletal Muscle Function
Methods: "A total of 468 subjects were randomized to atorvastatin or placebo of whom 203 in the atorvastatin and 214 in the placbo group provided complete data for analysis." The paper also includes a study flow diagram detailing the randomized and analyzed participants.