Monitoring✓ Mathematical
1 min read
◆ The PatternIs your model still valid? PSI, KS test, and detecting when the world has changed
Models decay when the data distribution shifts. PSI (Population Stability Index) detects changes in feature distributions. The KS test checks if two samples came from the same distribution. Monitor these to know when to retrain.
PSI = Σ (pᵢ − qᵢ) · ln(pᵢ / qᵢ)
PSI < 0.1 = no drift | 0.1–0.25 = moderate | >0.25 = significant shift
KS = max|Fref(x) − Fnew(x)|
Kolmogorov-Smirnov = maximum distance between two CDFs. p < 0.05 → distributions differ.
// Interactive — reference vs new distribution with drift
Drift amount0.0
PSI—
# Python — PSI & KS test from scipy.stats import ks_2samp import numpy as np def psi(ref, new, bins=10): edges = np.histogram_bin_edges(ref, bins=bins) p = np.histogram(ref, bins=edges)[0] / len(ref) + 1e-6 q = np.histogram(new, bins=edges)[0] / len(new) + 1e-6 return np.sum((p - q) * np.log(p / q)) # KS test stat, p_val = ks_2samp(ref_data, new_data) print(f"KS stat: {stat:.3f}, p: {p_val:.4f}")
Pattern bridge: Data drift in ML mirrors regime changes in markets. Both signal that the rules have changed — past patterns no longer predict the future.
Real-world pipeline: monitoring for drift
- Baseline: Save reference distributions from your training data (histograms, quantiles per feature)
- Schedule: Run PSI or KS test on each feature weekly (or per batch in streaming)
- Alert thresholds: PSI > 0.1 = moderate drift, PSI > 0.25 = significant — trigger investigation
- Investigate: Check if drift is in inputs (covariate shift) or in the target (concept drift)
- Retrain or adapt: If performance has degraded, retrain on recent data. For gradual drift, use a sliding training window
Common pitfall — feature drift ≠ model decay: A feature's distribution can shift without affecting model performance (if it's a low-importance feature). Always cross-check drift detection with actual model metrics on labelled data.
Performance in practice
- Uber's Michelangelo platform monitors 10K+ models — PSI is the primary drift signal. They retrain automatically when PSI > 0.2 on any top-10 feature
- COVID-19 caused massive covariate shift in credit scoring models — income, spending, employment features all drifted simultaneously. Models that flagged drift early saved banks millions in bad decisions
- Gradual drift (seasonal) is normal — build it into your retraining schedule. Sudden drift (COVID, regulatory change) requires immediate response
When to use this
✓ Use when: Any production ML model. The longer a model runs without monitoring, the more likely it has silently degraded. Especially critical for high-stakes decisions (lending, healthcare, fraud).
✗ Skip when: One-off analyses where the model won't be reused. Static datasets that never change (benchmark competitions). Models retrained on every batch already (online learning).
Try it on real data
Kaggle: Credit Card Fraud — split by time to simulate temporal drift
Split the dataset at the midpoint (Time column). Compute PSI between first-half and second-half feature distributions — you'll see real drift in V1-V28.
Quick start — copy to notebook
pip install scipy pandas numpy
import numpy as np
from scipy.stats import ks_2samp
def psi(ref, new, bins=10):
edges = np.histogram_bin_edges(ref, bins=bins)
p = np.histogram(ref, bins=edges)[0] / len(ref) + 1e-6
q = np.histogram(new, bins=edges)[0] / len(new) + 1e-6
return np.sum((p - q) * np.log(p / q))
feature = 'V14'
ref = train_df[feature].dropna().to_numpy()
new = production_df[feature].dropna().to_numpy()
ks_stat, p_value = ks_2samp(ref, new)
print({'psi': psi(ref, new), 'ks': ks_stat, 'p_value': p_value})