Time series and forecasting · insight-1211
Say whether a point forecast targets the mean or the median, and score it consistently
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- retail, aviation, cross-industry
Summary: situation, action, reason, boundary
Situation
A skewed demand model is trained on squared error but selected on MAE, and two teams argue over which candidate is more accurate.
Action
Decide which functional the decision needs (mean for totals, median or a quantile for service levels). Train and select on a score that is consistent for it: squared error for the mean, absolute or pinball loss for a quantile.
Reason
Scoring rules are consistent for specific functionals. Evaluating a mean forecast by absolute error, or a median by squared error, penalises the right answer for the wrong reason.
Boundary
Skewed or intermittent demand, revenue, durations, any series where mean and median differ.
Symptom or risk
The metrics reward different summaries of the same predictive distribution, so rankings flip with the metric and no single winner exists without a stated target.
Exceptions
For symmetric predictive distributions, the two coincide.
Trade-offs
Mean forecasts add up across items; medians do not, which matters for hierarchies.
Pitfalls
Reporting both RMSE and MAE for one model and treating a disagreement as noise.
Example
Hypothetical week of demand 0,0,0,5,5,5,60: forecast 5 (median) has MAE 10.0 and MSE 442.9; forecast 10.7 (mean) has MAE 14.1 and MSE 410.2.
Lessons and concepts
Claims and their sources
A scoring function consistent for the mean is a Bregman function (squared error is the standard example), and quantiles are elicitable with generalised piecewise-linear scores; mismatched scoring can give misleading comparisons.
Quantiles are optimal point forecasts under generalised piecewise linear loss, with the level set by the costs of under- and over-prediction.