Time series and forecasting · insight-1226
Monitor forecast error as a ratio to a frozen benchmark, not as an absolute number
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A weekly dashboard alerts when MAPE exceeds 12%, and it fires every holiday season and stays quiet through real model decay in calm months.
Action
Run the benchmark (seasonal naive, or the last accepted model frozen at a date) alongside production and monitor the ratio of errors, using a rolling window and a run-length rule for the alert.
Reason
The ratio cancels the difficulty common to both forecasts and isolates what the model adds. A frozen reference separates model decay from a harder world.
Boundary
Production forecast monitoring with delayed actuals.
Symptom or risk
Difficulty changes with season and volatility, so absolute thresholds produce false alarms in hard periods and miss decay in easy ones.
Exceptions
Where a contractual absolute accuracy target exists, track both.
Trade-offs
Needs the benchmark forecast logged at issue time; the benchmark itself can be damaged by a structural break.
Pitfalls
Letting the benchmark retrain, so it adapts with the model and masks drift.