Statistics✓ Mathematical
1 min read
◆ The PatternNull vs alternative, what a p-value actually means, and the two ways to be wrong
Every statistical claim follows the same script. Assume nothing interesting is happening (the null hypothesis H₀), collect data, and ask: if H₀ were true, how surprising would this data be? That surprise, quantified, is the p-value. If it drops below a pre-chosen threshold α (usually 0.05), you reject H₀ in favour of the alternative H₁.
p = P(data at least this extreme | H₀ true)
p-value = probability of seeing a test statistic this extreme assuming the null is true. It is NOT the probability that H₀ is true.
z = (x̄ − μ₀) / (σ / √n)
Test statistic = how many standard errors the observed mean sits from the null value. Big |z| → small p.
α = P(Type I) β = P(Type II) Power = 1 − β
Type I = false alarm (reject a true null) | Type II = miss (fail to reject a false null)
// Interactive — slide the observed statistic and α, watch the p-value and the decision flip
Observed z1.80
α0.05
p-value—
Decision—
| H₀ is true | H₀ is false | |
|---|---|---|
| Reject H₀ | Type I error (prob α) | Correct — power (1−β) |
| Fail to reject | Correct | Type II error (prob β) |
# Python — one-sample and two-sample tests from scipy import stats import numpy as np # Does this sample's mean differ from 100? sample = np.random.normal(103, 15, size=50) t_stat, p_val = stats.ttest_1samp(sample, popmean=100) print(f"t = {t_stat:.2f}, p = {p_val:.4f}") # Do two groups differ? t_stat, p_val = stats.ttest_ind(group_a, group_b) alpha = 0.05 if p_val < alpha: print("Reject H0 — difference is statistically significant") else: print("Fail to reject H0 — not enough evidence")
What a p-value is NOT: it is not the probability the null is true, not the probability the result is a fluke, and not a measure of effect size. p = 0.001 with a tiny, irrelevant effect is common with big data — always pair p-values with effect size.
Pattern bridge: This machinery powers comparing model runs and A/B tests, and power analysis tells you how much data the test needs before you run it.
How to use this in practice
- State H₀ and H₁ before looking at the data, and fix α up front
- Pick the test that matches your data — see choosing the right test
- Run the test, report the p-value and the effect size with a confidence interval
- Interpret: p < α means "surprising if nothing were going on" — not "large" or "important"
Common pitfall — p-hacking: testing many metrics, subgroups, or stopping rules until something crosses 0.05 guarantees false positives. With 20 independent tests at α = 0.05, you expect one significant result by pure chance. Pre-register your analysis or correct for multiple comparisons (Bonferroni, Benjamini-Hochberg).
When to use this
✓ Use when: Comparing groups, validating that a change moved a metric, checking whether a model improvement is real — any decision that should survive sampling noise.
✗ Skip when: You have the full population (no sampling uncertainty), or the question is "how big is the effect?" rather than "is there an effect?" — go straight to estimation with confidence intervals.
Try it on real data
Kaggle: Mobile Games A/B Testing — Cookie Cats (90K players)
UCI: Wine Quality (red vs white — a natural two-group comparison)
Cookie Cats is a real product experiment: did moving a gate from level 30 to 40 change retention? Perfect first hypothesis test.
Quick start — copy to notebook
pip install scipy numpy
# ────────────────────────────────────────
import numpy as np
from scipy import stats
rng = np.random.default_rng(7)
control = rng.normal(100, 15, 200)
variant = rng.normal(104, 15, 200)
t, p = stats.ttest_ind(variant, control, equal_var=False)
d = (variant.mean() - control.mean()) / np.sqrt(
(control.var() + variant.var()) / 2)
print(f"t = {t:.2f}, p = {p:.4f}, Cohen's d = {d:.2f}")