Time series and forecasting · insight-0218
Track realised interval coverage online and adjust the miscoverage level when it drifts
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- ML engineer, Data scientist
- Industries
- telecom, energy, saas
Summary: situation, action, reason, boundary
Situation
A deployed 90% interval for hourly call volumes covered 90% in backtests but 78% over the last month.
Action
Monitor rolling coverage per horizon, and use an online update such as adaptive conformal inference that raises the target coverage after misses and lowers it after hits.
Reason
Adaptive conformal inference was designed to keep long-run coverage close to target under distribution shift.
Boundary
Streaming forecasts with fast feedback: call centres, energy load, web traffic.
Symptom or risk
Staffing plans that rely on the upper bound are short-staffed repeatedly, and nobody notices because only point accuracy is monitored.
Exceptions
With slow feedback (monthly or quarterly), online adjustment reacts too late; recalibrate at each cycle instead.
Trade-offs
Adaptive intervals widen after shocks, which can look erratic to users; explain why.
Pitfalls
Calling long-run coverage a guarantee for each individual period.
Lessons and concepts
Claims and their sources
Adaptive conformal inference updates the miscoverage level online to maintain long-run coverage under distribution shift.