Hypothesis Testing & p-values

27 — Statistical Foundations

Hypothesis Testing & p-values#

StatisticsStatistical 1 min read

◆ The PatternNull vs alternative, what a p-value actually means, and the two ways to be wrong

Every statistical claim follows the same script. Assume nothing interesting is happening (the null hypothesis H₀), collect data, and ask: if H₀ were true, how surprising would this data be? That surprise, quantified, is the p-value. If it drops below a pre-chosen threshold α (usually 0.05), you reject H₀ in favour of the alternative H₁.

p = P(data at least this extreme | H₀ true)

p-value = probability of seeing a test statistic this extreme assuming the null is true. It is NOT the probability that H₀ is true.

z = (x̄ − μ₀) / (σ / √n)

Test statistic = how many standard errors the observed mean sits from the null value. Big |z| → small p.

α = P(Type I) β = P(Type II) Power = 1 − β

Type I = false alarm (reject a true null) | Type II = miss (fail to reject a false null)

// Interactive — slide the observed statistic and α, watch the p-value and the decision flip

Observed z1.80

α0.05

p-value—

Decision—

	H₀ is true	H₀ is false
Reject H₀	Type I error (prob α)	Correct — power (1−β)
Fail to reject	Correct	Type II error (prob β)

What a p-value is NOT: it is not the probability the null is true, not the probability the result is a fluke, and not a measure of effect size. p = 0.001 with a tiny, irrelevant effect is common with big data — always pair p-values with effect size.

Pattern bridge: This machinery powers comparing model runs and A/B tests, and power analysis tells you how much data the test needs before you run it.

How to use this in practice

State H₀ and H₁ before looking at the data, and fix α up front
Pick the test that matches your data — see choosing the right test
Run the test, report the p-value and the effect size with a confidence interval
Interpret: p < α means "surprising if nothing were going on" — not "large" or "important"

Common pitfall — p-hacking: testing many metrics, subgroups, or stopping rules until something crosses 0.05 guarantees false positives. With 20 independent tests at α = 0.05, you expect one significant result by pure chance. Pre-register your analysis or correct for multiple comparisons (Bonferroni, Benjamini-Hochberg).

When to use this

✓ Use when: Comparing groups, validating that a change moved a metric, checking whether a model improvement is real — any decision that should survive sampling noise.

✗ Skip when: You have the full population (no sampling uncertainty), or the question is "how big is the effect?" rather than "is there an effect?" — go straight to estimation with confidence intervals.

Try it on real data

Kaggle: Mobile Games A/B Testing — Cookie Cats (90K players) UCI: Wine Quality (red vs white — a natural two-group comparison)

Cookie Cats is a real product experiment: did moving a gate from level 30 to 40 change retention? Perfect first hypothesis test.

Quick start — copy to notebook

pip install scipy numpy
# ────────────────────────────────────────
import numpy as np
from scipy import stats

rng = np.random.default_rng(7)
control = rng.normal(100, 15, 200)
variant = rng.normal(104, 15, 200)

t, p = stats.ttest_ind(variant, control, equal_var=False)
d = (variant.mean() - control.mean()) / np.sqrt(
    (control.var() + variant.var()) / 2)
print(f"t = {t:.2f}, p = {p:.4f}, Cohen's d = {d:.2f}")

Worked example

5,000 experiments with no real effect: 5.0% come out significant, exactly the false-positive rate promised. 5,000 experiments with a real effect of 0.4 standard deviations and 30 per group: only 32.3% come out significant, and 18.2% land between 0.01 and 0.05. With low power, a real effect usually looks like nothing, and when it does show, it barely clears the line.

Where it misleads

Significant results from low-powered studies exaggerate the effect size and can even get its sign wrong (Gelman & Carlin 2014).
When most tested hypotheses are false, most significant findings can be false too (Ioannidis 2005).
p < 0.05 is a convention, not a law; report the estimate and its interval.

In code

from scipy.stats import ttest_ind
rng = np.random.default_rng(18)
x = rng.normal(0, 1, (5_000, 60))                        # 5,000 experiments, two groups of 30, no effect
y = rng.normal(0, 1, (5_000, 60)); y[:, 30:] += 0.4      # 5,000 more with a real effect of 0.4 SD
null = ttest_ind(x[:, :30], x[:, 30:], axis=1).pvalue    # one t-test per row
real = ttest_ind(y[:, :30], y[:, 30:], axis=1).pvalue
share = (round((null < 0.05).mean(), 3), round((real < 0.05).mean(), 3))
near_miss = round(((real > 0.01) & (real < 0.05)).mean(), 3)

Python, in your browser — edit the code and run it again

Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.

The same idea with a library

# Python — one-sample and two-sample tests
from scipy import stats
import numpy as np

# Does this sample's mean differ from 100?
sample = np.random.normal(103, 15, size=50)
t_stat, p_val = stats.ttest_1samp(sample, popmean=100)
print(f"t = {t_stat:.2f}, p = {p_val:.4f}")

# Do two groups differ?
t_stat, p_val = stats.ttest_ind(group_a, group_b)

alpha = 0.05
if p_val < alpha:
    print("Reject H0 — difference is statistically significant")
else:
    print("Fail to reject H0 — not enough evidence")

Sources

R. L. Wasserstein & N. A. Lazar, “The ASA Statement on p-Values: Context, Process, and Purpose”, The American Statistician 70(2), 2016
J. P. A. Ioannidis, “Why Most Published Research Findings Are False”, PLoS Medicine 2(8), 2005
A. Gelman & J. Carlin, “Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors”, Perspectives on Psychological Science 9(6), 2014

Last reviewed October 2026 · How this site checks its content · Report a mistake

Check yourself

1. A test gives p = 0.03. Which statement is correct?

Show answer

B. A p-value is computed assuming the null is true, so it cannot be the probability that the null is true. It also says nothing about size: a tiny effect in a huge sample can give a tiny p-value.

2. You test 20 independent metrics at α = 0.05, and none of them is really affected. How likely is at least one “significant” result?

Show answer

A. Each test has a 95% chance of staying quiet, so all twenty stay quiet with probability 0.95²⁰ ≈ 0.36. The other 64% of the time something crosses the line by chance — the garden of forking paths.

3. A study reports p = 0.20 for a new treatment. What can you conclude?

Show answer

C. Failing to reject is not proof of the null. With a small sample (low power) a real effect often gives p > 0.05; report the effect size and its confidence interval instead.

Linked from 9

← Previous

Power Analysis

Choosing the Right Statistical Test

Open in the full reader, with the topic sidebar