Time series and forecasting · insight-1225
Evaluate reconciled forecasts level by level against the base forecasts, since coherence is not accuracy
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- retail, aviation, public-sector
Summary: situation, action, reason, boundary
Situation
A reconciled forecast sums exactly and the team ships it on the strength of a better headline number at the network level.
Action
In the same rolling backtest compare base and reconciled forecasts per level and per segment with scaled errors and distribution scores. Ship the variant that serves each consumer's level.
Reason
Reconciliation shares information across the hierarchy, but gains are not guaranteed everywhere and the hierarchy's scale differences mean errors must be compared within levels.
Boundary
Any hierarchical or grouped forecast with several consumers (network, region, route).
Symptom or risk
Reconciliation can improve aggregate levels while degrading some lower-level series, and exact coherence in-sample says nothing about out-of-sample error.
Exceptions
If coherence is a hard requirement (a single plan), accept the trade and monitor the worst level.
Trade-offs
Using different forecasts per consumer reintroduces incoherence.
Pitfalls
Averaging the error across levels into one number.
Example
In the prison-population example in the textbook, MinT had the best total-level MASE but a negative CRPS skill score at the bottom level (-3.23).
Lessons and concepts
Claims and their sources
In the textbook's prison example MinT had the best total-level MASE (0.90) while its bottom-level CRPS skill score was negative (-3.23); error measures are compared within each level because of scale differences.