Engineering, production, governance, cost, communication · insight-0270
Canary a new model on a small traffic slice with pre-agreed rollback criteria
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A retrained model is ready, and the plan is to switch all traffic at once on Monday.
Action
Send a small, time-limited share of traffic to the new model, compare it against the control on a few attributable, user-visible metrics and model-specific checks (score distribution, decision rate), and roll back automatically when thresholds are crossed.
Reason
Canarying exposes a change to a subset first and compares it with a control, limiting damage and making the go or no-go decision explicit.
Boundary
Online model deployments, rules engines, prompt and model-version changes in LLM services.
Symptom or risk
A bad model affects every user before monitoring notices, and a rollback decision becomes a debate during the incident.
Exceptions
Batch models can canary by segment or region instead of traffic share.
Trade-offs
Small canaries detect only large effects quickly; subtle quality changes need longer tests.
Pitfalls
Choosing canary metrics that react slower than the canary period.
Lessons and concepts
Claims and their sources
Canarying deploys a change to a small, time-limited part of production and compares it with a control; if the canary looks bad, the rollout pauses or rolls back.