A confidence interval gives a range of plausible values for an unknown parameter. A 95% CI means: if we repeated this experiment many times, 95% of the intervals computed this way would contain the true value. Width = uncertainty.
A t-interval from 10 observations is promised to cover the true mean 95% of the time. With normal data it does: 95.0% in 20,000 simulations. With skewed (lognormal) data it covers only 83.6%. The nominal level is a promise conditional on assumptions; small, skewed samples break it.
- Bootstrap intervals help with skew but need enough data themselves; BCa intervals correct for bias and skew (Efron 1987).
- An interval for the mean says nothing about where individual values will fall; that is a prediction interval.
- Overlapping intervals are not a significance test for a difference.
from scipy import stats
rng = np.random.default_rng(14)
def coverage(draw, true_mean, n=10, reps=20_000):
x = draw((reps, n))
half = stats.t.ppf(0.975, n - 1) * x.std(axis=1, ddof=1) / np.sqrt(n)
return (np.abs(x.mean(axis=1) - true_mean) <= half).mean()
normal = coverage(lambda s: rng.normal(1, 1, s), 1)
skewed = coverage(lambda s: rng.lognormal(0, 1, s), np.exp(0.5)) # lognormal, mean e^0.5Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.
The same idea with a library
# Python — confidence intervals import numpy as np from scipy.stats import sem, t # Parametric (t-distribution for small samples) n = len(data) ci = t.interval(0.95, df=n-1, loc=np.mean(data), scale=sem(data)) # Bootstrap boots = [np.mean(np.random.choice(data, size=n, replace=True)) for _ in range(10000)] ci_boot = np.percentile(boots, [2.5, 97.5])
- J. Neyman, “Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability”, Philosophical Transactions of the Royal Society A 236, 1937
- B. Efron, “Better Bootstrap Confidence Intervals”, Journal of the American Statistical Association 82(397), 1987
- Introduction to Robust Estimation and Hypothesis Testing (4th ed.), R. R. Wilcox, Academic Press, 2017
Last reviewed October 2026 · How this site checks its content · Report a mistake
1. A 95% confidence interval for a mean is [4.1, 5.3]. What does the “95%” describe?
Show answer
B. The 95% belongs to the procedure, not to one interval: this one either contains the true mean or it does not. The probability reading in C is what a Bayesian credible interval gives. A describes the spread of the data, which is a different, much wider thing.
2. You collect four times as much data. Roughly what happens to the width of the interval?
Show answer
B. The width scales with 1/√n, so four times the data halves it. Halving it again takes four times as much data again — precision gets expensive.
3. The 95% intervals for two groups overlap a little. Is the difference between them therefore not significant?
Show answer
A. Checking whether two intervals overlap is a much stricter test than p < 0.05; intervals can overlap by a fair amount while the difference is significant. Build an interval for the difference instead.
- Pattern EssaysThe Bell in EverythingWherever many small, independent forces combine, the same shape appears
- Chart PatternsChannelsTwo parallel trendlines containing price
- Chart PatternsDescending TriangleFlat support + falling highs = sellers pressing lower
- Technical IndicatorsIchimoku CloudMidpoints of past highs and lows, some shifted forward in time, to show balance, trend and support at a glance
- Technical IndicatorsKeltner ChannelsBands set by average range instead of standard deviation, so a single outlier moves them less
- Market PsychologyOverconfidenceBelieving you know more than you do.
- Risk & PortfolioValue at Risk (VaR)Maximum expected loss at a chosen confidence level over a given horizon
- ML MathBias-Variance TradeoffThe fundamental tension between underfitting and overfitting
- MLOps & Production MLAlerting & SLOsLatency budgets, error budgets, and avoiding alert fatigue
- MLOps & Production MLFairness AuditsTesting for bias across protected groups
- The ToolkitHypothesis Testing & p-valuesNull vs alternative, what a p-value actually means, and the two ways to be wrong