Time series and forecasting · insight-1227
Log the action taken on each forecast so that monitored error is not contaminated by your own response
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- Data scientist, ML engineer
- Industries
- aviation, retail, logistics
Summary: situation, action, reason, boundary
Situation
A capacity forecast triggers extra flights on routes forecast high, and the monitored error on those routes keeps improving.
Action
Store, with each forecast, the decision it fed (price change, capacity change, promotion) and its timing. Evaluate on untreated cohorts, or keep a small randomised holdout where the forecast is not acted on.
Reason
A forecast used for decisions is part of the system it predicts. Without a record of the intervention, observed error mixes model error and decision effect.
Boundary
Pricing, replenishment and capacity forecasts that feed automated actions.
Symptom or risk
Actions change the outcome the forecast is scored against, so accuracy can look better or worse for reasons that are not model quality, and decay stays hidden behind interventions.
Exceptions
Forecasts that do not drive actions (reporting) have no such loop.
Trade-offs
A holdout costs the value of the decision on those units.
Pitfalls
Reading improved accuracy after a rollout as a model improvement.