Time series and forecasting · insight-1244
Weight a global model's loss by the weights of the metric that will judge it
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- Data scientist, ML engineer
- Industries
- retail, e-commerce
Summary: situation, action, reason, boundary
Situation
A global boosted model for 30,000 SKU series is trained with unweighted squared error, but it will be judged on an error that is weighted by recent revenue.
Action
Scale each target by its series' error scale, set sample weights proportional to the metric's weights, computed from data before the origin, and report both the weighted metric and an unweighted tail view so low-value series are not abandoned.
Reason
A loss that mirrors the scoring weights aims at the quantity being judged. In M5 the score weights came from dollar sales over the last 28 training observations, with all 12 aggregation levels equally weighted.
Boundary
Global models for many series when value, revenue or margin differ by orders of magnitude.
Symptom or risk
Each training row counts equally while the metric does not, so capacity is spent where it scores little and the metric's heavy series get no extra attention.
Exceptions
If the business objective is per-series service level, equal weighting may be right.
Trade-offs
Weighted training can overfit the top series and degrade the long tail, which may still matter operationally.
Pitfalls
Computing weights from the evaluation window, which leaks information about which series will matter.
Lessons and concepts
Claims and their sources
The M5 accuracy competition ranked methods by a weighted RMSSE, with weights from cumulative dollar sales over the last 28 observations of the training sample and with 12 aggregation levels equally weighted.