Time series and forecasting · insight-0222
Score a quantile forecast with pinball loss at that quantile
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- retail, energy, logistics
Summary: situation, action, reason, boundary
Situation
A team forecasts the 0.9 quantile of demand and reports its MAPE against actuals.
Action
Evaluate each quantile with the quantile (pinball) score at its level and check the empirical hit rate; use CRPS when the whole distribution matters.
Reason
The quantile score is a proper scoring rule for quantile forecasts, so it rewards an honest quantile rather than a central estimate.
Boundary
Quantile regression, quantile gradient boosting, probabilistic deep forecasting models.
Symptom or risk
A 0.9 quantile should exceed actuals about 90% of the time, so point metrics punish it for doing its job and push the team toward the median.
Exceptions
If downstream users only consume a point forecast, evaluate that point separately.
Trade-offs
Pinball loss is unfamiliar to stakeholders; translate it into cost units.
Pitfalls
Averaging pinball scores across quantiles with different business weights.
Lessons and concepts
Claims and their sources
The quantile score (pinball loss) evaluates quantile forecasts, and CRPS averages quantile scores over all probability levels.
Strictly proper scoring rules reward honest probabilistic forecasts.