Time series and forecasting · insight-0242
Treat public-benchmark scores of forecasting foundation models as possibly contaminated
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A time-series foundation model is proposed because it tops a public leaderboard.
Action
Check whether the benchmark separates pretraining data from test data, prefer leaderboards that flag leakage, and run your own rolling-origin backtest on data the model cannot have seen, such as recent internal periods.
Reason
GIFT-Eval provides a non-leaking pretraining split and flags test-data leakage because overlap between pretraining and evaluation data is a recognised risk.
Boundary
Chronos, TimesFM, Moirai, TimeGPT and similar models.
Symptom or risk
Pretraining corpora may contain the benchmark series or close relatives, so zero-shot scores can reflect memorisation rather than generalisation to your data.
Exceptions
Proprietary series created after a model's training cutoff are safe test data by construction.
Trade-offs
Your own backtest costs time but is the only evidence on your series.
Pitfalls
Quoting a leaderboard rank without its date and evaluation settings.
Lessons and concepts
Claims and their sources
GIFT-Eval provides a non-leaking pretraining dataset split and evaluates statistical, deep learning and foundation models across many datasets, frequencies and horizons.
The GIFT-Eval leaderboard reports a test-data-leakage flag for submitted models.