MLOps & Production ML

Champion at 90% accuracy, challenger at 91%, both scoring the same labelled requests. McNemar’s test looks only at the cases where they disagree. After 2,000 requests the p-value is 0.65 — no evidence either way. After 20,000 it is 0.0009. Shadow mode is safe, but a one-point improvement needs a lot of traffic to show.

• Shadow mode only sees the champion’s world: if the challenger’s decisions would change user behaviour, the shadow comparison cannot see it.
• Comparing on the requests that happen to get labels quickly can bias the result.
• A shadow model doubles serving cost while it runs; set an end date.

from scipy.stats import binomtest
rng = np.random.default_rng(3)

def shadow(n, p_champ=0.90, p_chal=0.91):           # both models score the same n labelled requests
    champ = rng.random(n) < p_champ
    chal = np.where(rng.random(n) < 0.5, champ, rng.random(n) < p_chal)   # they agree half the time by construction
    only_chal, only_champ = (chal & ~champ).sum(), (champ & ~chal).sum()
    return binomtest(only_chal, only_chal + only_champ).pvalue           # McNemar's exact test

p_2k, p_20k = shadow(2_000), shadow(20_000)

Assumes import numpy as np and import pandas as pd. Each snippet carries its own example numbers; the comments say which are made up or simulated.