Time series and forecasting · insight-1223
Compare two forecasts with a paired test on per-origin loss differences
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Model B has a 3% lower mean error than model A over 52 weekly origins and is proposed as the replacement.
Action
Form the loss difference per origin (same series, same origin, same horizon), and test whether its mean is zero with a Diebold-Mariano-type statistic that allows serial correlation. Report the difference with an interval, not a p-value alone.
Reason
Pairing removes the shared difficulty of each origin, and a covariance-robust variance accounts for correlated errors across overlapping multi-step forecasts.
Boundary
Model swaps and champion-challenger decisions on a rolling backtest.
Symptom or risk
Two averages are compared by eye, while per-origin losses are noisy and autocorrelated, so the 3% may be within the noise.
Exceptions
Very short backtests have no power; say so rather than testing.
Trade-offs
The test assumes a covariance-stationary loss differential; use caution around structural breaks.
Pitfalls
Running many pairwise tests across dozens of models without a multiplicity plan.
Lessons and concepts
Claims and their sources
Diebold and Mariano propose tests of equal predictive accuracy that allow non-quadratic and asymmetric loss and forecast errors that are non-Gaussian, nonzero mean, serially and contemporaneously correlated.