Significance✓ Mathematical
◆ The PatternStatistically significant ≠ meaningful — Cohen's d and the difference between p-values and impact
With enough data, any tiny difference becomes statistically significant. Effect size measures how big the difference is, independent of sample size. Cohen's d expresses the difference in standard deviation units.
d = (μ1 − μ2) / spooled
Cohen's d: 0.2 = small, 0.5 = medium, 0.8 = large. Tells you the magnitude of the effect.
// Interactive — two distributions, see effect size and overlap
Mean difference0.50
Cohen's d—
Overlap %—
# Python — effect size import numpy as np def cohens_d(group1, group2): n1, n2 = len(group1), len(group2) var1, var2 = group1.var(), group2.var() pooled_std = np.sqrt(((n1-1)*var1 + (n2-1)*var2) / (n1+n2-2)) return (group1.mean() - group2.mean()) / pooled_std d = cohens_d(model_a_scores, model_b_scores) print(f"Cohen's d = {d:.3f}")
Always report both: "The improvement was statistically significant (p = 0.02) with a medium effect size (d = 0.55)." p-value alone is meaningless.