Engineering, production, governance, cost, communication · insight-1264
Set canary duration from event volume and cycle length, not from a clock
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- ML engineer, Data scientist
- Industries
- aviation, e-commerce, cross-industry
Summary: situation, action, reason, boundary
Situation
A canary for a booking-conversion model runs for 30 minutes at 5% of traffic, and the rollout then goes to 100% on 'no issues'.
Action
Estimate how many decisions in each key segment the canary needs to detect a change you care about, cover at least one full load cycle if load matters, and extend the stage until both are met.
Reason
A canary that sees only a handful of requests gives no usable signal, and defects often appear only under peak load or in smaller segments.
Boundary
Staged rollouts of models, features and thresholds where the guardrail metric is an event rate.
Symptom or risk
A rare segment or a daily peak never appeared in the window, so 'nothing wrong' meant 'nothing observed'; the defect lands on everyone.
Exceptions
For crash-class failures a short, small first stage is the right design; size the later stages for statistical detection.
Trade-offs
Longer canaries slow release cadence and expose more users to a possible defect.
Pitfalls
Comparing canary metrics to the previous hour instead of to a concurrent control.
Example
Hypothetical: 5% of 40,000 daily fare requests is 2,000; a segment that is 2% of traffic yields about 40 requests, too few to see a 2 point change in a rate.
Lessons and concepts
Claims and their sources
The SRE Workbook chapter on canarying says a canary that handles very few requests gives no useful signal, that it must run long enough and see representative traffic, and that before and after comparisons are noisy because of time-of-day effects.