Time series and forecasting · insight-0202
Report ex-ante accuracy, not ex-post, when a model uses future-unknown regressors
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- energy, retail, aviation
Summary: situation, action, reason, boundary
Situation
A team reports that adding temperature cut daily electricity-load forecast error by a third.
Action
Split regressors into future-known (calendar, holidays, scheduled capacity) and future-unknown (weather, competitor actions). Evaluate with the forecasts of the unknown ones that existed at each origin, ideally archived vintages.
Reason
Ex-ante forecasts use only information available in advance and are the genuine forecasts. Ex-post results are useful for diagnosing the model, but they are not the accuracy users will experience.
Boundary
Energy, retail promotions, airline demand with competitor fares, any dynamic regression.
Symptom or risk
The backtest used observed temperatures for the target days. That is an ex-post evaluation, so it says how well the model explains load, not how well it forecasts it.
Exceptions
If the regressor will be set by your own decision (your price, your capacity), conditioning on planned values is legitimate scenario forecasting.
Trade-offs
Archiving weather or competitor forecasts costs storage and engineering, but without them you cannot audit real accuracy.
Pitfalls
Reconstructing old forecasts from today's revised data understates the error.
Lessons and concepts
Claims and their sources
Ex-ante forecasts use only information available in advance and are genuine forecasts; ex-post forecasts use later information on the predictors and are useful for studying model behaviour.