Time series and forecasting · insight-0217
Check exchangeability before trusting split-conformal coverage on a time series
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist, ML engineer
- Industries
- energy, retail, aviation
Summary: situation, action, reason, boundary
Situation
Split conformal intervals are calibrated on last year's residuals and promised to give 90% coverage.
Action
Treat the 90% as an assumption to test: backtest coverage by period and horizon, calibrate on recent residuals per horizon, and consider weighted or adaptive conformal methods when the series drifts.
Reason
The finite-sample guarantee of split conformal prediction relies on exchangeable calibration and test data, and the guarantee is marginal, not per period.
Boundary
Any conformal interval on temporal data, including ML forecasts and demand models.
Symptom or risk
Temporal dependence and drift break the exchangeability assumption, so realised coverage can fall well below 90% after a regime shift.
Exceptions
Residuals from a well-specified model on a stable series can be approximately exchangeable; verify with backtests.
Trade-offs
Recent-window calibration adapts faster but uses fewer residuals, so intervals are noisier.
Pitfalls
Calibrating on in-sample residuals, which are too small.
Lessons and concepts
Claims and their sources
Split conformal prediction gives finite-sample marginal coverage when calibration and test data are exchangeable.
Weighted conformal methods bound the coverage gap when exchangeability fails, as with drift or time series.