MLOps & Production ML

To catch an error rate rising from 1.0% to 1.5% with the usual 5% significance and 80% power takes about 7,747 requests in each arm. With 100,000 requests a day and the canary on 5%, that is 1.5 days of canary traffic. A ten-minute canary on 1% of traffic can only catch outright breakage, not a regression of this size.

• Peeking at the canary every few minutes and stopping when it looks fine inflates false confidence; decide the sample size or use a sequential test.
• Canary traffic is often not a random slice (one region, one client version), so the comparison can be confounded.
• Business metrics such as conversion move slowly; a canary judged only on latency and errors can still ship a worse model.

from scipy.stats import norm
p0, p1 = 0.010, 0.015                  # error rate today, and the regression we want to catch
alpha, power = 0.05, 0.80
z = norm.ppf(1 - alpha / 2) + norm.ppf(power)
n = z**2 * (p0 * (1 - p0) + p1 * (1 - p1)) / (p1 - p0) ** 2    # requests needed in each arm
daily = 100_000
days_at_5pct = n / (0.05 * daily)       # canary gets 5% of traffic

Assumes import numpy as np and import pandas as pd. Each snippet carries its own example numbers; the comments say which are made up or simulated.

# Kubernetes canary with Istio traffic split
# VirtualService:
#   http:
#   - route:
#     - destination:
#         host: model-service
#         subset: stable
#       weight: 95
#     - destination:
#         host: model-service
#         subset: canary
#       weight: 5