Classical ML, validation, metrics, interpretation, debugging · insight-1714
Recalibrate an average of calibrated models, because averaging probabilities makes the blend underconfident
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- finance, cross-industry
Summary: situation, action, reason, boundary
Situation
Models that each pass a reliability check are blended by averaging their probabilities, and the blend feeds an expected-value or threshold rule.
Action
Treat the blend as a new model: fit a recalibration (a logistic map on its logit, or isotonic with plenty of rows) on held-out rows after blending, check the reliability curve, and version it with the weights.
Reason
Ranjan and Gneiting show that any non-trivial weighted average of distinct calibrated probability forecasts is necessarily uncalibrated and lacks sharpness.
Boundary
Probability-space averaging of model versions and of different model types.
Symptom or risk
The average of distinct calibrated forecasts is itself uncalibrated: pulled to the middle, so high scores are not high enough.
Exceptions
Near-identical members change little; a logistic stacker already recalibrates.
Trade-offs
One more fitted step; with few rows prefer the logistic map to isotonic.
Pitfalls
Skipping it because each member passed.
Example
Simulated (30% positives): two forecasters read independent Gaussian signals of one outcome, calibrated by construction (slope 1.00). Their average has slope 2.5; where it predicts 0.65 the outcome rate is 0.93. Log loss: 0.321 for one forecaster, 0.260 averaged, 0.192 recalibrated. At signal correlation 0.9 the slope is 1.09.
Lessons and concepts
Claims and their sources
Ranjan and Gneiting's abstract states that any non-trivial weighted average of two or more distinct calibrated probability forecasts is necessarily uncalibrated and lacks sharpness, so linear pooling requires recalibration even when members are calibrated, and proposes a beta-transformed linear opinion pool.