Engineering, production, governance, cost, communication · insight-1267
Alert on error-budget burn rate over two windows for model service objectives
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A model endpoint has a latency or valid-response objective, and the on-call is paged by a fixed threshold that either fires on every blip or misses a slow degradation.
Action
Define the objective and its budget over a rolling period, page when the burn rate is high over a long window and still high over a short one, and send slow burns to a ticket queue instead.
Reason
Burn-rate rules scale urgency with how fast the budget is consumed, and the short confirming window makes the alert reset soon after recovery.
Boundary
Services with a stated availability, latency or valid-prediction objective.
Symptom or risk
Fixed thresholds page on blips yet miss a steady 3 percent error rate that would exhaust the monthly budget in days.
Exceptions
Very low-traffic services lack the volume for stable rates; use a count-based rule or a synthetic probe.
Trade-offs
More rules and windows to tune and to explain to the team.
Pitfalls
Using the same alert for model quality, where labels arrive late and the rate cannot be computed in real time.
Example
Hypothetical: for a 99.9% objective, page at 14.4x burn over one hour confirmed by five minutes; ticket at 1x over three days.
Lessons and concepts
Claims and their sources
The SRE Workbook recommends multiwindow, multi-burn-rate alerts, for example a page at 14.4x burn over 1 hour confirmed by 5 minutes and a ticket at 1x burn over 3 days for a 99.9% SLO.