Time series and forecasting · insight-1222

Replay intermittent-demand forecasts through a stock policy and report service and stock, not only error

Situation
Spare-parts demand is mostly zeros with occasional orders, and two methods tie on MASE.
Action
Simulate an order-up-to rule driven by each method's predictive quantile over the lead time on the backtest history. Report fill rate, stock-out events and average stock held, and compare at equal service.
Reason
Inventory outcomes depend on the distribution of lead-time demand, so a decision replay measures what the metric cannot.
Boundary
Slow movers and intermittent series feeding stock or capacity decisions.

Symptom or risk

Point-error metrics reward forecasts near zero and do not show stock-outs or excess; equal error can mean very different service and inventory.

If no stock decision follows (reporting only), distribution scores are enough.

A replay needs cost and lead-time assumptions and a policy; results are conditional on them.

Replaying with actual demand known at ordering time.

Kolassa argues that MAD, MASE and wMAPE are inherently unsuitable for count data and recommends evaluating entire discrete predictive distributions with proper scoring rules.
Evaluating predictive count data distributions in retail sales forecasting