14 — Analyze Your Data

Missing Data Strategies#

Data QualityStatistical 1 min read
◆ The PatternImputation, MICE, missingness patterns — handling gaps without corrupting your analysis

Missing data isn't just annoying — why it's missing matters. MCAR (completely random) is safe to drop. MAR (depends on observed data) needs smart imputation. MNAR (depends on the missing value itself) is the hardest case.

MCAR: P(missing) = constant
Missing completely at random. Dropping rows is unbiased but wasteful.
MAR: P(missing | observed) ≠ P(missing)
Missing at random given observed data. Use MICE, KNN imputation.
// Interactive — missing data patterns and imputation
% Missing20%
Strategy
Never impute the target. And always impute inside cross-validation folds to prevent data leakage.
Pattern bridge: Gaps are rarely random: a missing price can mean a halted stock, a missing reading a failed sensor. Handling them is part of data quality in production and of resampling irregular time series.
Worked example

10,000 incomes with a true mean of 43,741, where higher earners skip the question more often (17% missing overall). The mean of the answers is 36,509 — and imputing the mean for the gaps keeps exactly that bias, while also shrinking the spread. When missingness depends on the value itself, no method on the observed data alone can fully recover it.

Where it misleads
  • Missing not at random (MNAR) cannot be detected from the data; it needs domain knowledge or a sensitivity analysis.
  • Multiple imputation handles missing-at-random well; it does not fix MNAR.
  • A missingness indicator can itself be predictive, and dropping it throws information away.
In code
rng = np.random.default_rng(9)
income = rng.lognormal(10.5, 0.6, 10_000)
p_missing = np.clip((income - 30_000) / 100_000, 0, 0.8)     # higher earners skip the question more
observed = np.where(rng.random(10_000) < p_missing, np.nan, income)
true_mean = income.mean()
complete_case = np.nanmean(observed)                          # what mean imputation also centres on
share_missing = np.isnan(observed).mean()
Python, in your browser — edit the code and run it again

Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.

The same idea with a library
# Python — imputation strategies
from sklearn.impute import SimpleImputer, KNNImputer
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer

# Simple: median (robust to outliers)
imp = SimpleImputer(strategy='median')

# KNN: uses similar rows
imp = KNNImputer(n_neighbors=5)

# MICE: iterative multivariate
imp = IterativeImputer(max_iter=10)
X_filled = imp.fit_transform(X)
Sources
  • D. B. Rubin, “Inference and Missing Data”, Biometrika 63(3), 1976
  • Statistical Analysis with Missing Data (3rd ed.), R. J. A. Little & D. B. Rubin, Wiley, 2019
  • S. van Buuren & K. Groothuis-Oudshoorn, “mice: Multivariate Imputation by Chained Equations in R”, Journal of Statistical Software 45(3), 2011

Last reviewed October 2026 · How this site checks its content · Report a mistake

Linked from 2
← Previous
Outlier Detection
Open in the full reader, with the topic sidebar