22 — Make Decisions

Confidence Intervals#

InferenceStatistical 1 min read
◆ The PatternHow sure are you? — bootstrap and parametric CIs, and interpreting their width

A confidence interval gives a range of plausible values for an unknown parameter. A 95% CI means: if we repeated this experiment many times, 95% of the intervals computed this way would contain the true value. Width = uncertainty.

CI = x̄ ± z · (σ / √n)
Parametric CI for means. z = 1.96 for 95%. Width shrinks with √n.
Bootstrap CI: [θ*2.5%, θ*97.5%]
Bootstrap CI = percentiles of the resampled distribution. No normality assumption needed.
// Interactive — sample size vs CI width
Sample size50
Confidence95%
Width—
Pattern bridge: CIs quantify uncertainty. In ML, report metric ± CI from cross-validation. In markets, Monte Carlo confidence bands are CIs for portfolio outcomes.
Worked example

A t-interval from 10 observations is promised to cover the true mean 95% of the time. With normal data it does: 95.0% in 20,000 simulations. With skewed (lognormal) data it covers only 83.6%. The nominal level is a promise conditional on assumptions; small, skewed samples break it.

Where it misleads
  • Bootstrap intervals help with skew but need enough data themselves; BCa intervals correct for bias and skew (Efron 1987).
  • An interval for the mean says nothing about where individual values will fall; that is a prediction interval.
  • Overlapping intervals are not a significance test for a difference.
In code
from scipy import stats
rng = np.random.default_rng(14)
def coverage(draw, true_mean, n=10, reps=20_000):
    x = draw((reps, n))
    half = stats.t.ppf(0.975, n - 1) * x.std(axis=1, ddof=1) / np.sqrt(n)
    return (np.abs(x.mean(axis=1) - true_mean) <= half).mean()
normal = coverage(lambda s: rng.normal(1, 1, s), 1)
skewed = coverage(lambda s: rng.lognormal(0, 1, s), np.exp(0.5))   # lognormal, mean e^0.5
Python, in your browser — edit the code and run it again

Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.

The same idea with a library
# Python — confidence intervals
import numpy as np
from scipy.stats import sem, t

# Parametric (t-distribution for small samples)
n = len(data)
ci = t.interval(0.95, df=n-1, loc=np.mean(data), scale=sem(data))

# Bootstrap
boots = [np.mean(np.random.choice(data, size=n, replace=True))
         for _ in range(10000)]
ci_boot = np.percentile(boots, [2.5, 97.5])
Sources
  • J. Neyman, “Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability”, Philosophical Transactions of the Royal Society A 236, 1937
  • B. Efron, “Better Bootstrap Confidence Intervals”, Journal of the American Statistical Association 82(397), 1987
  • Introduction to Robust Estimation and Hypothesis Testing (4th ed.), R. R. Wilcox, Academic Press, 2017

Last reviewed October 2026 · How this site checks its content · Report a mistake

Check yourself

1. A 95% confidence interval for a mean is [4.1, 5.3]. What does the “95%” describe?

Show answer

B. The 95% belongs to the procedure, not to one interval: this one either contains the true mean or it does not. The probability reading in C is what a Bayesian credible interval gives. A describes the spread of the data, which is a different, much wider thing.

2. You collect four times as much data. Roughly what happens to the width of the interval?

Show answer

B. The width scales with 1/√n, so four times the data halves it. Halving it again takes four times as much data again — precision gets expensive.

3. The 95% intervals for two groups overlap a little. Is the difference between them therefore not significant?

Show answer

A. Checking whether two intervals overlap is a much stricter test than p < 0.05; intervals can overlap by a fair amount while the difference is significant. Build an interval for the difference instead.

Linked from 11
← Previous
Survivorship & Look-Ahead Bias
Open in the full reader, with the topic sidebar