ML CI/CD extends traditional CI/CD with three additional test layers: data validation (schema + quality), model validation (performance thresholds), and serving validation (latency + correctness). A merge should trigger retraining, evaluation, and conditional deployment.
| Stage | Tests | Gate |
|---|---|---|
| Data | Schema, freshness, completeness | All checks pass |
| Training | Convergence, no NaN loss | Loss below threshold |
| Evaluation | AUC, F1, latency benchmark | Metrics ≥ champion |
| Deployment | Smoke test, shadow run | No errors in canary |
A candidate model beats the champion overall, 0.906 against 0.898 accuracy on 20,000 holdout cases. But on the 1,000 enterprise customers it is 4 points worse. The gate requires that no segment loses more than one point, so the candidate is blocked. An overall metric would have shipped it.
- Slices with few examples are noisy; a gate on them needs a tolerance or a confidence interval, or it blocks at random.
- A fixed holdout used for every gate is slowly overfitted by repeated decisions against it; refresh it.
- Tests on the model alone miss pipeline bugs; test the data and the serving path too.
holdout = pd.DataFrame({'segment': ['new', 'returning', 'enterprise'],
'n': [8_000, 11_000, 1_000],
'champion': [0.880, 0.910, 0.900],
'candidate': [0.895, 0.918, 0.860]}) # accuracy per segment
overall = lambda col: (holdout[col] * holdout.n).sum() / holdout.n.sum()
better_overall = overall('candidate') > overall('champion')
worst_drop = (holdout.candidate - holdout.champion).min()
ship = better_overall and worst_drop >= -0.01 # gate: no segment may lose more than one pointAssumes import numpy as np and import pandas as pd. Each snippet carries its own example numbers; the comments say which are made up or simulated.
- E. Breck, S. Cai, E. Nielsen, M. Salib & D. Sculley, “The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction”, IEEE International Conference on Big Data, 2017
- D. Sato, A. Wider & C. Windheuser, “Continuous Delivery for Machine Learning”, martinfowler.com, 2019
- D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems”, Advances in Neural Information Processing Systems 28, 2015
Last reviewed October 2026 · How this site checks its content · Report a mistake