Operations✓ Mathematical
◆ The PatternWhen models fail in production — rollback, fallback, postmortem
ML incidents are different from software bugs: the code runs fine, but predictions are wrong. Rollback reverts to the previous model version. Fallbacks (rule-based defaults, cached predictions) serve something when the model is down. Circuit breakers automatically switch to fallback when error rates spike.
// Interactive — incident response decision tree
| Severity | Response | Timeline |
|---|---|---|
| P0 — Model serving errors | Rollback immediately | Minutes |
| P1 — Accuracy degradation | Switch to fallback, investigate | Hours |
| P2 — Slight drift detected | Schedule retraining | Days |
| P3 — Feature quality warning | Monitor and log | Next sprint |
Blameless postmortems: After every incident, document what happened, why detection was delayed, and what systemic fix prevents recurrence. Blame the system, not the person.
Pattern bridge: ML incident response mirrors stop-loss discipline in trading — predefined rules that limit damage when things go wrong. The power analysis framework helps design monitoring that catches problems early enough to act.