Missing data isn't just annoying — why it's missing matters. MCAR (completely random) is safe to drop. MAR (depends on observed data) needs smart imputation. MNAR (depends on the missing value itself) is the hardest case.
10,000 incomes with a true mean of 43,741, where higher earners skip the question more often (17% missing overall). The mean of the answers is 36,509 — and imputing the mean for the gaps keeps exactly that bias, while also shrinking the spread. When missingness depends on the value itself, no method on the observed data alone can fully recover it.
- Missing not at random (MNAR) cannot be detected from the data; it needs domain knowledge or a sensitivity analysis.
- Multiple imputation handles missing-at-random well; it does not fix MNAR.
- A missingness indicator can itself be predictive, and dropping it throws information away.
rng = np.random.default_rng(9) income = rng.lognormal(10.5, 0.6, 10_000) p_missing = np.clip((income - 30_000) / 100_000, 0, 0.8) # higher earners skip the question more observed = np.where(rng.random(10_000) < p_missing, np.nan, income) true_mean = income.mean() complete_case = np.nanmean(observed) # what mean imputation also centres on share_missing = np.isnan(observed).mean()
Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.
The same idea with a library
# Python — imputation strategies from sklearn.impute import SimpleImputer, KNNImputer from sklearn.experimental import enable_iterative_imputer from sklearn.impute import IterativeImputer # Simple: median (robust to outliers) imp = SimpleImputer(strategy='median') # KNN: uses similar rows imp = KNNImputer(n_neighbors=5) # MICE: iterative multivariate imp = IterativeImputer(max_iter=10) X_filled = imp.fit_transform(X)
- D. B. Rubin, “Inference and Missing Data”, Biometrika 63(3), 1976
- Statistical Analysis with Missing Data (3rd ed.), R. J. A. Little & D. B. Rubin, Wiley, 2019
- S. van Buuren & K. Groothuis-Oudshoorn, “mice: Multivariate Imputation by Chained Equations in R”, Journal of Statistical Software 45(3), 2011
Last reviewed October 2026 · How this site checks its content · Report a mistake