Classical ML, validation, metrics, interpretation, debugging · insight-0148
Use sigmoid calibration on small calibration sets and isotonic only with plenty of data
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You choose a calibration method for a classifier with a modest validation set.
Action
Default to sigmoid (Platt) below roughly a thousand calibration samples, try isotonic above that, and always calibrate on data disjoint from the model's training data (for example CalibratedClassifierCV).
Reason
Isotonic is non-parametric and flexible, which needs data; sigmoid has two parameters and is stable on small samples.
Boundary
Binary classifiers whose probabilities will be used, in scikit-learn or equivalent tooling.
Symptom or risk
Isotonic regression overfits a few hundred points into a jagged step function and can change ROC-AUC through ties.
Exceptions
If the miscalibration is not sigmoid-shaped and data are ample, isotonic or spline calibration can win clearly.
Trade-offs
Cross-validated calibration uses data efficiently but fits several models; a single held-out calibration split is simpler.
Pitfalls
Calibrating on the same rows the model was trained on, which makes the calibrator learn overconfident training scores.
Lessons and concepts
Claims and their sources
Isotonic calibration is more prone to overfitting and is best with more than about 1,000 samples; calibration data must be disjoint from training data; isotonic introduces ties that can change ROC-AUC.