Time series and forecasting · insight-1210
Aggregate ratios of errors to a benchmark with a geometric mean, not an arithmetic mean
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- retail, cross-industry
Summary: situation, action, reason, boundary
Situation
Model A is compared with seasonal naive on 500 SKUs by averaging each SKU's error ratio (model error divided by benchmark error).
Action
Compute the per-series error ratio, take the geometric mean across series, and report the share of series where the ratio is below one next to it.
Reason
Under the geometric mean, a halving and a doubling cancel, so over- and under-performance are treated evenly. The arithmetic mean is dominated by the worst ratios.
Boundary
Comparing methods over many series with different scales, including forecast value added summaries.
Symptom or risk
Ratios are bounded below by zero and unbounded above, so one SKU at 3.0 outweighs ten at 0.5 and a model that is better on most series looks worse.
Exceptions
If one pooled, scale-aware total is the business KPI, use that total error instead.
Trade-offs
Ratio measures are undefined when the benchmark error is zero; guard those series.
Pitfalls
Averaging ratios of already-scaled errors across horizons and series in one step.
Example
Hypothetical two-series case: ratios 0.5 and 2.0 average to 1.25 arithmetically (looks worse than the benchmark) and to 1.0 geometrically (no net gain).
Lessons and concepts
Claims and their sources
Davydenko and Fildes recommend aggregating performance ratios across series with the weighted geometric mean, which they describe as treating over- and under-forecasting evenly.