Time series and forecasting · insight-1221
Do not measure bias on uncensored days only, because dropping stock-outs selects low-demand days
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- Data scientist
- Industries
- aviation, retail
Summary: situation, action, reason, boundary
Situation
To avoid censoring, a team computes forecast bias only on days that did not sell out and reports a persistent over-forecast of 10%.
Action
Estimate bias with a censoring-aware method: compare the capped forecast with sales, use a model-based expected demand over censored days, or check against an external signal such as stock-out flags and denied booking counts.
Reason
Conditioning on an event that depends on the outcome changes the outcome distribution. The same logic that makes stock-out zeros misleading for training makes filtered bias misleading for evaluation.
Boundary
Any metric computed only on non-stock-out, non-capacity-bound or unconstrained observations.
Symptom or risk
Selecting on not sold-out means selecting days with lower demand, so the filtered mean demand is below the unconditional one and an unbiased forecast looks high.
Exceptions
If censoring is rare and independent of demand level (data loss), filtering is harmless.
Trade-offs
Censoring-aware estimates need a model of the tail.
Pitfalls
Adding a stock-out flag as a regressor and then dropping flagged rows from scoring.
Example
Hypothetical demand normal (100, 20), cap 110, forecast exactly 100: the mean over non-sold-out days is 89.8, so the forecast appears 11% too high.