MLOps & Production ML

Labels arrive weeks late, so watch the predictions. Over two baseline weeks 8.1% of 5,000 daily predictions are positive; a p-chart puts the control limits ±1.16 points away. When the true rate moves to 11% on day 20, the chart flags day 20 itself, and days 21 and 22 after it — weeks before any accuracy number could.

• A stable prediction rate does not prove a healthy model; the inputs and the outcome can change together.
• Watching dozens of charts at three-sigma limits produces false alarms; decide which signals page and which only inform.
• Dashboards nobody looks at are not monitoring. Tie each panel to an owner and an action.

rng = np.random.default_rng(2)
n = 5_000                                                 # predictions per day
rate = np.r_[np.full(20, 0.08), np.full(10, 0.11)]        # share predicted positive; it shifts on day 20
pos = rng.binomial(n, rate) / n
base = pos[:14].mean()                                    # two weeks of baseline
limit = 3 * np.sqrt(base * (1 - base) / n)                # p-chart control limits
flagged = np.where(np.abs(pos - base) > limit)[0]

Assumes import numpy as np and import pandas as pd. Each snippet carries its own example numbers; the comments say which are made up or simulated.