S14 · paper simulation

Check a forecasting foundation-model claim

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

5stages
40 minabout
0stages committed

The brief

A simulated colleague writes: "Foundation models now top the GIFT-Eval leaderboard. We should replace our ETS route forecasts with zero-shot TimesFM or Chronos and save the maintenance."

What is documented in the course source registry (check the records before quoting anything else):

GIFT-Eval is a benchmark spanning many datasets, frequencies and horizons, with a pretraining split designed not to leak test data (source: paper-aksu-2024-gift-eval). Its leaderboard repository logs added a test-data-leakage column and a MASE rank column (2025-08-05), a zero-shot tag and a replication-code column; positions change, so any ranking must be quoted with a date (source: repo-gift-eval).
Chronos and TimesFM are pretrained forecasters used zero-shot (sources: paper-ansari-2024-chronos, paper-das-2024-timesfm). Benchmark claims in vendor reports and blogs for newer versions are self-reported (sources: paper-ansari-2025-chronos-2, blog-google-2026-timesfm-3).
Simple linear models outperformed several Transformer long-horizon forecasters on standard benchmarks in one peer-reviewed study (source: paper-zeng-2023-dlinear).

Your use case: daily passengers for 24 routes of the synthetic airline, horizons 1-60 days, used for capacity and fare planning. Competitor capacity on three routes steps up 25% on 2024-03-01 (a known structural break in the synthetic data).

Stage 1 of 5

You open the leaderboard. You do not quote positions here; the question is what you need to know about the benchmark before its ranking can inform your decision.

Which questions must you answer before the claim is relevant to your use case? Select all that apply.

Is the entry flagged for possible test-data leakage, and is it tagged zero-shot or fine-tuned?Which metric and aggregation is the ranking based on (for example MASE or CRPS rank), and on which date?Do the benchmark's frequencies, horizons and domains resemble daily airline demand at 1-60 days?How many parameters does the model have, since larger models generalise better to unseen series and horizons?Is the ranking self-reported by the model's authors or produced by the benchmark maintainers with code?