StatisticsStatistical
1 min read
◆ The Patternt-test, chi-square, ANOVA and their nonparametric cousins — a decision map
Most real questions reduce to three: how many groups am I comparing, are the measurements paired, and can I assume normality? Answer those and the test picks itself. Parametric tests (t-test, ANOVA) assume roughly normal data and are more powerful; nonparametric tests (Mann-Whitney, Wilcoxon, Kruskal-Wallis) rank the data instead and work almost anywhere.
t = (x̄₁ − x̄₂) / √(s₁²/n₁ + s₂²/n₂)
Two-sample t (Welch) = difference in means scaled by its standard error. Default choice for two independent groups.
χ² = Σ (observed − expected)² / expected
Chi-square = for counts and categories — is the observed table compatible with independence?
// Interactive decision map — set your situation, the recommended test lights up
Groups2
PairedNo
Normal-ishYes
Use—
| Question | Parametric | Nonparametric |
|---|---|---|
| 2 independent groups differ? | Welch t-test | Mann-Whitney U |
| Before vs after (paired)? | Paired t-test | Wilcoxon signed-rank |
| 3+ groups differ? | One-way ANOVA | Kruskal-Wallis |
| Two categorical variables related? | Chi-square test of independence | |
| Two numeric variables related? | Pearson r | Spearman ρ |
| Sample from this distribution? | Kolmogorov-Smirnov / Shapiro-Wilk | |
Normality worries less than you think: with n > ~30 per group, the Central Limit Theorem makes the t-test robust to non-normal data. Reach for nonparametric tests when samples are small, heavily skewed, or ordinal.
ANOVA says "some group differs" — not which. Follow a significant ANOVA with pairwise post-hoc tests (Tukey HSD) rather than running raw t-tests between every pair, which inflates the false-positive rate.
When to use this
✓ Use when: You have a concrete comparison question and need the defensible test — experiments, feature launches, model comparisons, survey analysis.
✗ Skip when: The dataset is the full population, or you need effect estimates rather than yes/no answers — estimate with bootstrap or regression instead.
Try it on real data
Palmer Penguins (3 species — made for ANOVA and chi-square)
UCI: Wine Quality (rating groups, skewed features)
Penguins ships with seaborn (sns.load_dataset('penguins')) — three species, numeric and categorical columns, a few NaNs. Every test in the table above has a natural question here.
Quick start — copy to notebook
pip install scipy pandas seaborn
# ────────────────────────────────────────
import pandas as pd
import seaborn as sns
from scipy import stats
df = sns.load_dataset('penguins').dropna()
groups = [g for _, g in df.groupby('species')['body_mass_g']]
print(stats.f_oneway(*groups)) # 3 groups, parametric
print(stats.kruskal(*groups)) # 3 groups, rank-based
chi2, p, dof, _ = stats.chi2_contingency(
pd.crosstab(df['species'], df['island']))
print(f"chi2 = {chi2:.1f}, p = {p:.4g}")Pattern bridge: Choosing a test is choosing your assumptions. Drift detection uses the same tests — KS, chi-square — to ask whether production data still looks like training data, and stationarity tests ask whether a series keeps its statistics over time. Guide: Which statistical test should I use?
Worked example
Two groups of 40 from a heavy-tailed distribution (Student t with 2 degrees of freedom), with a real shift of 0.5. The t-test detects it 18% of the time; the Mann–Whitney test 35% — nearly double. With heavy tails a few extreme values swamp the mean, while ranks barely notice them. The right test depends on the shape of the data, not only on its type.
Where it misleads
- Mann–Whitney tests whether one group tends to be larger, not whether the means or medians differ, unless the shapes are the same.
- With large samples the t-test is robust to non-normality of the data; the problem is heavy tails and outliers, not skew alone.
- Running several tests and reporting the one that worked is a forking path.
In code
from scipy.stats import ttest_ind, mannwhitneyu
rng = np.random.default_rng(19)
hits_t = hits_u = 0
for _ in range(2_000): # heavy-tailed data with a real shift of 0.5
a, b = rng.standard_t(2, 40), rng.standard_t(2, 40) + 0.5
hits_t += ttest_ind(a, b).pvalue < 0.05
hits_u += mannwhitneyu(a, b).pvalue < 0.05
power = (hits_t / 2_000, hits_u / 2_000)Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.
The same idea with a library
# Python — the whole decision map in scipy from scipy import stats # 2 independent groups stats.ttest_ind(a, b, equal_var=False) # Welch t-test stats.mannwhitneyu(a, b) # nonparametric # Paired (same subjects, before/after) stats.ttest_rel(before, after) stats.wilcoxon(before, after) # nonparametric # 3+ groups stats.f_oneway(g1, g2, g3) # ANOVA stats.kruskal(g1, g2, g3) # nonparametric # Categorical vs categorical chi2, p, dof, exp = stats.chi2_contingency(contingency_table) # Check normality first (n < ~50) stat, p = stats.shapiro(a)
Sources
- H. B. Mann & D. R. Whitney, “On a Test of Whether One of Two Random Variables Is Stochastically Larger than the Other”, Annals of Mathematical Statistics 18(1), 1947
- M. W. Fagerland, “t-Tests, Non-Parametric Tests, and Large Studies — a Paradox of Statistical Practice?”, BMC Medical Research Methodology 12, 2012
- Introduction to Robust Estimation and Hypothesis Testing (4th ed.), R. R. Wilcox, Academic Press, 2017
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 2