Time series and forecasting · insight-1239
Train on the driver forecasts that existed at each origin, not on the values that were later observed
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- Data scientist, ML engineer
- Industries
- energy, aviation, logistics
Summary: situation, action, reason, boundary
Situation
A day-ahead load model is trained on observed temperature, but in production it receives a weather forecast whose error grows with lead time.
Action
Archive each driver forecast with its issue time and lead time, build each training row from the forecast issued at that origin for the target time, and compare it with the model trained on actuals in a backtest that uses the archived vintages.
Reason
Inputs that carry realistic noise teach the model how much to trust them, while realised values remove the very noise it will face in serving.
Boundary
Weather, macro and competitor-capacity drivers that are predicted rather than known at the forecast origin.
Symptom or risk
The model treats the driver as exact and leans on it too hard, so live error exceeds backtest error and does not degrade gracefully as the weather forecast gets worse.
Exceptions
Calendar and schedule drivers that are known exactly need no vintage.
Trade-offs
Archived vintages may only exist for a short history; one workaround is to train on actuals with noise injected at the measured forecast-error level, which should itself be backtested.
Pitfalls
Using the latest revised forecast instead of the one issued before the origin, which reintroduces look-ahead.