Classical ML, validation, metrics, interpretation, debugging · insight-1149
Check calibration inside each segment, because overall calibration can hide offsetting errors
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- aviation, banking, insurance
Summary: situation, action, reason, boundary
Situation
The reliability diagram over all rows hugs the diagonal and the model is declared calibrated.
Action
Compare mean predicted and observed rates, and a reliability diagram, per route, channel or customer segment, with counts and intervals, and recalibrate per segment only where enough data exist.
Reason
Calibration in aggregate is an average of segment-level biases that can cancel.
Boundary
Probabilities used in expected-value decisions made separately per segment.
Symptom or risk
One market is over-predicted and another under-predicted by similar amounts; expected-cost decisions and capacity plans per segment are biased in opposite directions.
Exceptions
If decisions are made only at the portfolio level, aggregate calibration is the relevant check.
Trade-offs
Per-segment recalibration needs enough events per segment; otherwise pool or shrink.
Pitfalls
Reading wiggles in small segments as miscalibration without intervals.
Example
Hypothetical: two equal-sized routes; predicted no-show 12% overall and observed 12%, but 9% predicted against 6% observed on one route and 15% against 18% on the other.