Testing✓ Mathematical
◆ The PatternIs this improvement real? Statistical tests for comparing model performance across runs
Model A got 0.87 F1. Model B got 0.89. Is that difference real or just noise? You need statistical tests on paired cross-validation scores to answer confidently. Without them, you're just reading tea leaves.
t = (μA − μB) / SE(μA − μB)
Paired t-test on k-fold scores — the simplest comparison. Assumes normality of differences.
McNemar: χ² = (b − c)² / (b + c)
McNemar’s test — compares errors on the same test set. b,c = discordant predictions.
// Interactive — two model score distributions, see p-value
Model A mean0.85
Model B mean0.88
p-value—
# Python — comparing two models from scipy.stats import ttest_rel, wilcoxon # Paired t-test on k-fold scores scores_a = cross_val_score(model_a, X, y, cv=10) scores_b = cross_val_score(model_b, X, y, cv=10) t_stat, p_val = ttest_rel(scores_a, scores_b) # Non-parametric alternative w_stat, p_val = wilcoxon(scores_a, scores_b) print(f"p = {p_val:.4f}")
Rule of thumb: If p < 0.05, the difference is statistically significant — but also check effect size. A significant p-value with tiny effect size means the improvement is real but may not matter.