Time series and forecasting · insight-1219

Divide CRPS or interval scores by a benchmark's score before averaging across series

Situation
A portfolio of 300 routes reports mean CRPS per model, and the average is driven by five large trunk routes.
Action
For each series compute the ratio of the model's score to a benchmark's (naive or seasonal naive); aggregate the ratios with a geometric mean (skill = 1 minus that mean) and show the distribution and the volume-weighted view separately.
Reason
Skill scores remove units and express improvement relative to a benchmark; averaging ratios treats each series as one observation of relative performance.
Boundary
Probabilistic accuracy across series of different volume.

Symptom or risk

Scores carry the units of the series, so the mean ranks models by how they do on the biggest series, not across the portfolio.

If the business KPI is total absolute cost, the unscaled sum is the right aggregate.

A weak benchmark inflates skill; a near-perfect benchmark makes skill volatile with small test sets.

Using a benchmark that is itself badly calibrated.

A skill score measures the relative improvement of a method over a benchmark, and it needs a large enough test set for a reliable denominator.
Forecasting: Principles and Practice (3rd edition)