Time series and forecasting · insight-1228
Report how many series the new model beats the benchmark on, next to the average improvement
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Data scientist, Data analyst, ML engineer
- Industries
- retail, cross-industry
Summary: situation, action, reason, boundary
Situation
A candidate model improves average MASE over 2,000 series by 4% and the team proposes a full cutover.
Action
Report the share of series where the model beats the benchmark, the median and tail of the per-series change, and the worst segments. Set a rule such as a maximum share or size of degradations before cutover.
Reason
Averages hide the distribution. Decision makers own series, so the count and severity of regressions is the risk.
Boundary
Migrations of forecasting systems and champion-challenger rollouts over many series.
Symptom or risk
A mean improvement can come from a few series with large gains while many series degrade, and a cutover quietly worsens whole segments.
Exceptions
A single total is the only consumer, so aggregate accuracy is the criterion.
Trade-offs
Per-series differences are noisy on short test sets; combine with a paired test.
Pitfalls
Declaring a win-rate of 55% a clear gain without checking it against noise.