15 — Analyze Your Data

Data Drift & Distribution Shift#

MonitoringStatistical 1 min read
◆ The PatternIs your model still valid? PSI, KS test, and detecting when the world has changed

Models decay when the data distribution shifts. PSI (Population Stability Index) detects changes in feature distributions. The KS test checks if two samples came from the same distribution. Monitor these to know when to retrain.

PSI = Σ (pᵢ − qᵢ) · ln(pᵢ / qᵢ)
PSI < 0.1 = no drift  |  0.1–0.25 = moderate  |  >0.25 = significant shift
KS = max|Fref(x) − Fnew(x)|
Kolmogorov-Smirnov = maximum distance between two CDFs. p < 0.05 → distributions differ.
// Interactive — reference vs new distribution with drift
Drift amount0.0
PSI—
Pattern bridge: Data drift in ML mirrors regime changes in markets. Both signal that the rules have changed — past patterns no longer predict the future.
Real-world pipeline: monitoring for drift
  1. Baseline: Save reference distributions from your training data (histograms, quantiles per feature)
  2. Schedule: Run PSI or KS test on each feature weekly (or per batch in streaming)
  3. Alert thresholds: PSI > 0.1 = moderate drift, PSI > 0.25 = significant — trigger investigation
  4. Investigate: Check if drift is in inputs (covariate shift) or in the target (concept drift)
  5. Retrain or adapt: If performance has degraded, retrain on recent data. For gradual drift, use a sliding training window
Common pitfall — feature drift ≠ model decay: A feature's distribution can shift without affecting model performance (if it's a low-importance feature). Always cross-check drift detection with actual model metrics on labelled data.
Performance in practice
  • Platforms such as Uber’s Michelangelo track feature and prediction distributions for every deployed model, and alert or retrain when they shift
  • COVID-19 shifted income, spending and employment features at once, and many credit and demand models degraded within weeks — drift can be sudden and hit every feature together
  • Gradual (seasonal) drift is normal — build it into the retraining schedule. Sudden drift (a pandemic, a regulatory change) needs an immediate response
When to use this
✓ Use when: Any production ML model. The longer a model runs without monitoring, the more likely it has silently degraded. Especially critical for high-stakes decisions (lending, healthcare, fraud).
✗ Skip when: One-off analyses where the model won't be reused. Static datasets that never change (benchmark competitions). Models retrained on every batch already (online learning).
Try it on real data
Kaggle: Credit Card Fraud — split by time to simulate temporal drift
Split the dataset at the midpoint (Time column). Compute PSI between first-half and second-half feature distributions — you'll see real drift in V1-V28.
Quick start — copy to notebook
pip install scipy pandas numpy
import numpy as np
from scipy.stats import ks_2samp

def psi(ref, new, bins=10):
    edges = np.histogram_bin_edges(ref, bins=bins)
    p = np.histogram(ref, bins=edges)[0] / len(ref) + 1e-6
    q = np.histogram(new, bins=edges)[0] / len(new) + 1e-6
    return np.sum((p - q) * np.log(p / q))

feature = 'V14'
ref = train_df[feature].dropna().to_numpy()
new = production_df[feature].dropna().to_numpy()
ks_stat, p_value = ks_2samp(ref, new)
print({'psi': psi(ref, new), 'ks': ks_stat, 'p_value': p_value})
Worked example

A classifier that catches 80% of positives with 10% false alarms, unchanged. When 10% of cases are positive its precision is 0.471; when 20% are, 0.667. Nothing about the features’ distribution within each class moved — only the share of positives (label shift) — yet a key metric changed by 20 points.

Where it misleads
  • Monitoring input features alone misses label shift and concept drift; track outcomes when they arrive.
  • Covariate shift, label shift and concept shift need different fixes; diagnose which one you have first (Moreno-Torres et al. 2012).
  • Thresholds tuned under one prevalence are wrong under another; re-tune when the base rate moves.
In code
tpr, fpr = 0.80, 0.10                               # the model's behaviour within each class, unchanged
def precision(prevalence):
    return tpr * prevalence / (tpr * prevalence + fpr * (1 - prevalence))
before, after = precision(0.10), precision(0.20)       # only the share of positives changed
Python, in your browser — edit the code and run it again

Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.

The same idea with a library
# Python — PSI & KS test
from scipy.stats import ks_2samp
import numpy as np

def psi(ref, new, bins=10):
    edges = np.histogram_bin_edges(ref, bins=bins)
    p = np.histogram(ref, bins=edges)[0] / len(ref) + 1e-6
    q = np.histogram(new, bins=edges)[0] / len(new) + 1e-6
    return np.sum((p - q) * np.log(p / q))

# KS test
stat, p_val = ks_2samp(ref_data, new_data)
print(f"KS stat: {stat:.3f}, p: {p_val:.4f}")
Sources
  • J. G. Moreno-Torres, T. Raeder, R. Alaiz-Rodríguez, N. V. Chawla & F. Herrera, “A Unifying View on Dataset Shift in Classification”, Pattern Recognition 45(1), 2012
  • Z. C. Lipton, Y.-X. Wang & A. Smola, “Detecting and Correcting for Label Shift with Black Box Predictors”, ICML, 2018
  • Dataset Shift in Machine Learning, J. Quiñonero-Candela, M. Sugiyama, A. Schwaighofer & N. D. Lawrence (eds.), MIT Press, 2009

Last reviewed October 2026 · How this site checks its content · Report a mistake

Linked from 2
← Previous
Missing Data Strategies
Open in the full reader, with the topic sidebar