Time series and forecasting · insight-1209
Retire sMAPE from forecast reports because of its asymmetry and its failure at zero
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Data scientist, Data analyst
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A report uses symmetric MAPE because it is bounded and the team believes it fixes MAPE's bias toward low forecasts.
Action
Replace it with a scaled error (MASE or RMSSE) for point accuracy and with WAPE for portfolio totals. If a percentage is mandatory, use MAPE only where actuals are far from zero and say so.
Reason
The symmetric denominator mixes the forecast into the scale, so the score is not a clean function of the error. The textbook advises against it.
Boundary
Cross-series comparison, competitions and dashboards still using M3-era metrics.
Symptom or risk
For the same absolute miss sMAPE punishes under-forecasts more than over-forecasts, and at an actual of zero every nonzero forecast scores the maximum, so model rankings become arbitrary.
Exceptions
Reproducing a published benchmark that reports sMAPE.
Trade-offs
Scaled errors are less intuitive to stakeholders than a percentage.
Pitfalls
Dropping zero-actual rows to make sMAPE computable, which removes the hardest cases.
Example
Actual 100: forecast 50 scores 66.7%, forecast 150 scores 40.0%. Actual 0: forecasts of 1 and 100 both score 200%.
Lessons and concepts
Claims and their sources
sMAPE was proposed to address the asymmetry of MAPE; the textbook authors advise against using it, noting instability near zero and that it can be negative.
Hyndman and Koehler critique percentage-based and relative measures and propose MASE, scaling errors by the in-sample naive MAE.