06 — Monitor & Observe

Drift Detection in Production#

MonitoringStatistical 1 min read
◆ The PatternDetecting when the world changes under your model

Data drift means input distributions have shifted. Concept drift means the relationship between inputs and targets has changed. Both silently degrade model performance. Continuous monitoring with PSI, KS tests, and page-based detection catches drift before users notice.

PSI = Σ (pi − qi) · ln(pi / qi)
PSI < 0.1 = stable  |  0.1–0.2 = moderate shift  |  > 0.2 = significant drift
// Interactive — reference vs production distributions
Drift Amount0.10
PSI—
Pattern bridge: Drift detection in ML is the same problem as data drift analysis in The Toolkit and regime detection in markets — the distribution has changed, and your old assumptions no longer hold.
Worked example

Five score bins, training shares against this week’s: the population stability index is 0.136. A common rule of thumb reads below 0.1 as stable and above 0.25 as a major shift, so this is “watch”. The second half of the snippet shows the opposite trap: a shift of just 0.02 standard deviations, irrelevant for any model, gives a KS-test p-value of 0.0000006 on 200,000 points.

Where it misleads
  • With big data every test is significant; judge the size of the shift (PSI, effect size), not the p-value.
  • Input drift is not performance drift: a model can be fine on shifted inputs and broken on unchanged ones (concept drift). Watch outcomes when labels arrive.
  • PSI depends on the binning; fix the bins at training time.
In code
from scipy.stats import ks_2samp
expected = np.array([0.10, 0.20, 0.40, 0.20, 0.10])     # share of training data in five score bins
actual   = np.array([0.05, 0.15, 0.35, 0.25, 0.20])     # the same bins this week
psi = np.sum((actual - expected) * np.log(actual / expected))

rng = np.random.default_rng(1)                           # a shift too small to matter...
train, live = rng.normal(0, 1, 200_000), rng.normal(0.02, 1, 200_000)
ks_p = ks_2samp(train, live).pvalue                      # ...is still "significant" with this much data
Python, in your browser — edit the code and run it again

Assumes import numpy as np and import pandas as pd. Each snippet carries its own example numbers; the comments say which are made up or simulated.

The same idea with a library
# Python — drift detection with Evidently
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset

report = Report(metrics=[DataDriftPreset()])
report.run(reference_data=df_train,
           current_data=df_prod)
report.save_html("drift_report.html")
Sources
  • N. Siddiqi, Credit Risk Scorecards, Wiley, 2006 — the PSI thresholds used in credit scoring
  • J. Gama, I. Žliobaitė, A. Bifet, M. Pechenizkiy & A. Bouchachia, “A Survey on Concept Drift Adaptation”, ACM Computing Surveys 46(4), 2014
  • S. Rabanser, S. Günnemann & Z. Lipton, “Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift”, NeurIPS, 2019

Last reviewed October 2026 · How this site checks its content · Report a mistake

Check yourself

1. A drift detector fires on the input data, but measured accuracy has not changed. What does that mean?

Show answer

A. Inputs can shift in ways the model handles well. Drift tells you where to look; outcomes tell you whether it matters.

2. On a million rows, a Kolmogorov–Smirnov test flags even tiny shifts as significant. What is better?

Show answer

A. With enough data every difference is significant. The question is whether it is large enough to matter.

3. What is concept drift?

Show answer

B. Fraud patterns adapting to a detector are a classic case. Input monitoring cannot see it; only labelled outcomes can.

Linked from 2
← Previous
GPU Inference
Open in the full reader, with the topic sidebar