Time series and forecasting · insight-1232
Compare models by AICc only within one model class and one differencing order
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist
- Industries
- cross-industry, retail
Summary: situation, action, reason, boundary
Situation
A team puts ETS, ARIMA and an ARIMA with an extra difference into one AICc table and ships the lowest value.
Action
Fix the differencing order with unit-root or seasonal-strength checks and a backtest, then use AICc only among models of the same class and the same differencing. Compare across classes, and against a seasonal naive and a boosted model, on out-of-sample error from identical origins.
Reason
FPP3 states that ETS and ARIMA are different model classes whose likelihoods are computed differently, and that changing the differencing order changes the data the likelihood is evaluated on.
Boundary
Automatic model selection for exponential smoothing and ARIMA families.
Symptom or risk
The likelihoods are computed in different ways or on differently differenced data, so the ranking is meaningless and may select a model that backtests worse.
Exceptions
Candidate sets that share class and differencing can be ranked by AICc as a screening step.
Trade-offs
Out-of-sample comparison costs more compute than reading a criterion column.
Pitfalls
Letting an automatic search silently vary d and then reporting its AICc as if it were comparable.
Lessons and concepts
Claims and their sources
FPP3 says AICc cannot be used to compare ARIMA and ETS models because they are in different classes with differently computed likelihoods, and that models compared by AICc must have the same orders of differencing.