Time series and forecasting · insight-0204
Use many rolling origins that span the seasons you care about, not one final holdout
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A forecast model is chosen from a single holdout of the last eight weeks before peak season.
Action
Run a rolling-origin backtest: move the cutoff forward step by step, train only on data before each cutoff, forecast the full horizon, and compare models on the distribution of errors across origins.
Reason
Averaging over many origins reduces the variance of the accuracy estimate and shows whether a model wins consistently or only in one window.
Boundary
Model selection and reporting for any production forecast.
Symptom or risk
One origin gives one noisy sample of accuracy, and it may sit entirely in an unrepresentative period, so the ranking of models can flip next month.
Exceptions
Very short histories may allow only a few origins; then say so and prefer simple models.
Trade-offs
More origins cost compute and refits; refitting every k origins is a common compromise.
Pitfalls
Tuning hyperparameters on the same origins you report; keep a later block untouched.
Lessons and concepts
Claims and their sources
Time-series cross-validation evaluates on a rolling forecasting origin, with each training set containing only observations that occurred before the test observations.
Tashman reviews how out-of-sample tests are designed, including fixed-origin versus rolling-origin procedures.