Time series and forecasting · insight-1207
Build the naive benchmark at the production horizon from the origin value, not from the previous day
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Data scientist, Data analyst, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A 28-day-ahead model is compared with a naive baseline computed as yesterday's actual for every target day.
Action
Generate the benchmark in the same loop as the model: the last value known at the origin carried forward h steps, or the seasonal value from the matching period at least h steps back, with the same refit and data cutoffs.
Reason
A naive forecast issued at the origin has an error whose standard deviation grows with h (about the square root of h under a random walk). Benchmarks must obey the model's information set.
Boundary
Every multi-step evaluation with a benchmark, including forecast value added.
Symptom or risk
The one-step naive uses information that does not exist at the origin, so it is far too strong at long horizons and the model's skill is understated or hidden.
Exceptions
If the production process really updates daily and only forecasts one day, the one-step naive is the right benchmark.
Trade-offs
A weaker benchmark makes the model look better, so state the construction in the report.
Pitfalls
Computing naive with a shift(1) on the target column of the test frame.
Example
Under a random walk the horizon-28 naive error is about 5.3 times the one-step error (square root of 28).
Lessons and concepts
Claims and their sources
For the naive method the h-step forecast standard deviation is the residual standard deviation times the square root of h; for seasonal naive it is that times the square root of the integer part of (h-1)/m plus one.