Time series and forecasting · insight-1215
Treat hits of overlapping multi-step intervals as dependent when testing coverage
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Intervals for a 14-day horizon are issued every day, and coverage is computed over 120 consecutive origins as if each hit were independent.
Action
Test hit sequences for clustering (a conditional-coverage or independence test), thin the origins to one per horizon length for the headline test, or use a block bootstrap, and report runs of consecutive misses.
Reason
A well-calibrated interval has hits that are both at the nominal rate and unpredictable from past hits. Dependence breaks the usual binomial reasoning.
Boundary
Rolling backtests and live coverage monitors with h greater than one origin spacing.
Symptom or risk
Adjacent origins share outcome days and shocks, so misses come in runs and the effective number of independent hits is far below 120; the confidence in a coverage verdict is overstated.
Exceptions
Non-overlapping origins (one per horizon) are close to independent.
Trade-offs
Thinning wastes data; block methods need a block length choice.
Pitfalls
Reporting overall coverage near nominal while all the misses sit in one disruption fortnight.
Lessons and concepts
Claims and their sources
Christoffersen argued that testing unconditional coverage alone is insufficient for evaluating interval forecasts and proposed a framework for conditional interval forecast evaluation.