project-023 · Forecasting · public-sector
Zero-shot time-series foundation models vs strong baselines on post-release monthly data (docs-only)
A labour-market planning unit forecasts monthly unemployment rates by country 1-12 months ahead for budget planning. A vendor proposes a pretrained time-series foundation model used zero-shot. Should the unit adopt it, given its accuracy relative to strong statistical baselines, its interval quality and its operating cost?
At a glance
Data
Eurostat une_rt_m - Unemployment by sex and age, monthly (public) · About 38 geographic entities in the latest period, monthly from as early as 1983-01 for some countries (Germany: 1983-01 to 2026-09 when checked).
Details on dataBaseline
Seasonal naive (12 months), AutoETS, AutoTheta and their median ensemble (statsforecast). The ensemble is the bar to clear; beating seasonal naive alone is not evidence.
Details on baselineValidation
Rolling monthly origins from the first month after the latest candidate release to the last available month, horizons 1-12, the same origins and context for every…
Details on validationMetric
MASE (seasonal period 12), scaled quantile loss averaged over 9 quantiles as a CRPS approximation, 80% interval coverage, win rate across countries.
Details on metric
Success criterion
Adopt only if, on rolling origins that all fall after the model's public release, the foundation model beats an ensemble of seasonal naive, AutoETS and AutoTheta on MASE and on scaled quantile loss, wins in more than half of the countries, and its 80% intervals cover between 75% and 85%. Otherwise the deliverable is a documented "not now" decision with the evidence.
Context
Foundation models such as Chronos and TimesFM are pretrained on large time-series corpora and forecast new series without training. Their published benchmark results cannot be transferred to a new task without checking for overlap between pretraining data and evaluation data: a public series released before the model's training cutoff may have been seen. GIFT-Eval addresses this with a non-leaking pretraining split; this project addresses it by evaluating only on months published after the model release. This project is docs-only: it requires PyTorch and model-weight downloads, so its code is checked against the model repositories' documentation, not executed in CI.
Data
- Dataset
- Eurostat une_rt_m - Unemployment by sex and age, monthly (public)
- Licence
- Reuse authorised provided the source is acknowledged (Eurostat copyright notice); data from non-Eurostat sources may restrict commercial reuse
- Size
- About 38 geographic entities in the latest period, monthly from as early as 1983-01 for some countries (Germany: 1983-01 to 2026-09 when checked).
- Verification
- identity verified
Access
Eurostat dissemination API (JSON or SDMX), no login, for example filtering sex = T, age = TOTAL, unit = PC_ACT, s_adj = NSA. Save the download date with the data, because values are revised between releases.
Grain
Country x month, unemployment rate as % of the labour force, not seasonally adjusted.
| Column | Meaning |
|---|---|
| geo | Country or aggregate (EU27, euro area); exclude aggregates from the country panel |
| time | Reference month (YYYY-MM) |
| s_adj | Seasonal adjustment; use NSA so seasonality is part of the task |
| unit | PC_ACT = percentage of the labour force |
| value | Unemployment rate (latest vintage; earlier releases may have differed) |
EDA and data-quality rules
Record the release date of each candidate model from its repository or paper; the first evaluation origin must be later than the latest release.
Exclude aggregates (EU27, euro area) and countries with gaps in the evaluation window.
Plot 2020-2021 for every country: the pandemic shock sits inside the context window and can distort zero-shot forecasts.
Store the download timestamp; re-downloading later changes history through revisions.
Baseline
Seasonal naive (12 months), AutoETS, AutoTheta and their median ensemble (statsforecast). The ensemble is the bar to clear; beating seasonal naive alone is not evidence.
Features
None for the zero-shot models beyond the context window (for example the last 120 months) and the horizon. Record context length, model size and any scaling the library applies; these are the "hyperparameters" of a zero-shot evaluation and must be fixed before looking at test months.
Candidate methods
Chronos (zero-shot, probabilistic) via the chronos-forecasting repository
Produces sample-based quantiles directly, so interval quality can be compared with the baselines.
TimesFM (zero-shot) via the timesfm repository
A second, differently trained family; if both lose to the ensemble, the conclusion is not specific to one model.
Validation design
Rolling monthly origins from the first month after the latest candidate release to the last available month, horizons 1-12, the same origins and context for every method. Report per-country and pooled metrics with a win/loss count, and a sign test across countries.
Leakage checks
Pretraining contamination: list what each model's documentation says about its training corpus and treat any public series older than the release as possibly seen; evaluate only post-release months. Revision leakage: latest-vintage values for early months were not known in real time; note it, because the unit's real forecasts would use first-release values.
Bias and representativeness
Bias per country size and per regime (rising vs falling unemployment); zero-shot models may revert to recent levels differently from ETS.
Robustness checks
Vary context length (60 vs 120 months) and model size; check that the conclusion survives excluding the countries with the largest pandemic shocks.
Technical metrics
MASE (seasonal period 12), scaled quantile loss averaged over 9 quantiles as a CRPS approximation, 80% interval coverage, win rate across countries.
Business interpretation
Planning error in percentage points at the 6- and 12-month horizons used for budgeting, plus runtime and infrastructure cost per monthly run (CPU vs GPU), which must be measured, not assumed.
Error slices
Horizon 1-3 vs 4-12 months.
Countries with high vs low volatility.
Origins near turning points.
Implementation steps
Download the panel, save the vintage date, select countries and freeze the evaluation design before running any model.
Run the statistical baselines and ensemble in the rolling loop (CPU, runnable).
Install the foundation-model packages in a separate environment following their repositories; record versions and release dates.
Run zero-shot forecasts with frozen settings on the same origins; save quantiles.
Score, run the sign test, and write the adopt / not-now decision with costs.
Reproduction
Baselines: CPU script in the main environment. Foundation models: separate documented environment with pinned versions and model ids; marked docs-only in the academy because it downloads weights and needs PyTorch.
Limitations
Post-release evaluation reduces but does not remove contamination: earlier months of the same series may be in the pretraining data.
Latest-vintage data overstate real-time accuracy for every method equally only if revisions are unrelated to the method, which is not guaranteed.
One indicator and one frequency; the decision does not generalise to daily operational data.
The post-release rule limits which model versions are testable: the panel ends 2026-09 when checked (2026-10), so a model released in late 2025 leaves roughly a year of origins and one released in 2026 leaves almost none. Pin versions released early enough to leave a useful number of origins (and record that they may be older than the vendor's current offer), or report the short window and its wide uncertainty.
Production and monitoring
If adopted: monthly batch after the Eurostat release, with the ensemble running in parallel as a challenger and automatic fallback when the foundation model's rolling MASE exceeds the ensemble's for three months. Pin model versions; a new model release is a new candidate that repeats this evaluation.
Interview questions
A vendor shows a leaderboard where their model wins. What do you check before trusting it for your data?
How do you evaluate a pretrained model whose training data may include your series?
Which baseline would you insist on, and why is seasonal naive alone not enough?
Postmortem and alternatives
Write it as a decision record: what was compared, on which months, what won and by how much, cost, and the trigger for re-evaluation (a new model release or a regime change). Alternative: use the foundation model only as one member of the ensemble.
Rubric3 criteria
| Criterion | Meets | Exceeds |
|---|---|---|
| Evaluation design | Post-release origins, frozen settings, same origins for all methods. | Pre-registers the design and decision rule before running any foundation model. |
| Baseline strength | Ensemble of statistical baselines included and reported per country. | Shows where the ensemble beats each foundation model and explains why. |
| Decision quality | Adopt / not-now decision with accuracy, interval and cost evidence. | Names the re-evaluation trigger and the fallback mechanism. |
References7
- Chronos: Learning the Language of Time Series
Chronos design and zero-shot probabilistic forecasting.
- A decoder-only foundation model for time-series forecasting
TimesFM design and zero-shot point forecasting.
- amazon-science/chronos-forecasting: Chronos: Pretrained Models for Time Series Forecasting
Installation, model ids and inference API for the docs-only code.
- google-research/timesfm: TimesFM (Time Series Foundation Model)
Installation and inference API for TimesFM.
- GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation
Leakage-aware benchmark design with a non-leaking pretraining split.
- Time Travel in LLMs: Tracing Data Contamination in Large Language Models
Contamination detection reasoning from LLM evaluation; analogous risk for pretrained forecasters.
- Forecasting: theory and practice
Benchmarks and combinations as strong baselines in forecasting practice.