Statistics✓ Mathematical
1 min read
◆ The PatternConfounders, spurious correlations, and trends that reverse when you split the data
Correlation measures whether two variables move together — nothing more. A strong r can come from X causing Y, Y causing X, a hidden confounder Z driving both, selection effects, or plain chance. Simpson's paradox is the most dramatic failure: a trend that holds in every subgroup can reverse when the groups are pooled.
r = cov(X, Y) / (σX · σY) ∈ [−1, 1]
Pearson r = linear co-movement only. r = 0 does not mean independent; r = 0.9 does not mean causal.
// Interactive Simpson's paradox — increase the confounder, watch the pooled trend flip against the groups
Confounder strength0.70
Pooled r—
Within-group r—
| Why X and Y correlate | Example | Antidote |
|---|---|---|
| X causes Y | Price cut → more sales | Randomized experiment confirms it |
| Y causes X | More police ↔ more crime (reverse) | Temporal ordering, natural experiments |
| Confounder Z | Ice cream ↔ drownings (summer) | Stratify or control for Z |
| Selection bias | Only survivors measured | Audit how the sample was formed |
| Chance | Any two trending series | Out-of-sample checks, corrections |
# Python — Simpson's paradox in three lines of groupby import pandas as pd # Pooled correlation says one thing… print(df[['dose', 'recovery']].corr().iloc[0, 1]) # …every subgroup says the opposite print(df.groupby('severity') .apply(lambda g: g['dose'].corr(g['recovery']))) # Always check candidate confounders before trusting a trend pd.crosstab(df['severity'], df['treatment'], normalize='index')
Which number is right? Neither, automatically. If severity drives both dose and recovery, the within-group trend is the honest one. The answer depends on the causal structure — what drives what — not on the arithmetic.
Pattern bridge: Feature correlation shows the same trap inside models, and survivorship bias is selection-driven correlation in backtests. The only clean escape is randomization — the logic behind A/B testing.
How to use this in practice
- Before trusting any correlation, list plausible confounders — time, size, cohort, seasonality are the usual suspects
- Recompute the relationship within each stratum of the confounder (groupby is enough)
- If pooled and stratified trends disagree, resolve it with causal knowledge, not with more data
- For decisions that matter, push for a randomized test — it severs every confounder at once
Common pitfall — controlling for everything: conditioning on a variable that sits between cause and effect (a mediator), or on a common effect (a collider), creates bias instead of removing it. Adding every column to a regression is not caution — think about the causal graph first.
When to use this
✓ Use when: Any observational comparison — dashboards, cohort metrics, "users who do X retain better" claims, feature-importance readings you are tempted to act on.
✗ Skip when: The data comes from a properly randomized experiment — randomization already balances confounders (but check the randomization actually held).
Try it on real data
The classic case: UC Berkeley admissions (Simpson's paradox)
Palmer Penguins (pooled vs per-species correlations flip sign)
In the penguins data, bill length vs bill depth correlates negatively pooled — and positively within every species. A Simpson's paradox you can groupby in one line.
Quick start — copy to notebook
pip install pandas seaborn
# ────────────────────────────────────────
import seaborn as sns
df = sns.load_dataset('penguins').dropna()
pooled = df['bill_length_mm'].corr(df['bill_depth_mm'])
within = df.groupby('species').apply(
lambda g: g['bill_length_mm'].corr(g['bill_depth_mm']))
print(f"pooled r = {pooled:.2f}") # negative!
print(within.round(2)) # all positive