Classical ML, validation, metrics, interpretation, debugging · insight-1150
Check ranking metrics after isotonic calibration, which turns scores into plateaus
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You add isotonic calibration to a model whose top-K list or AUC is used downstream.
Action
Compare AUC, precision at K and the number of distinct scores before and after calibration; use sigmoid calibration, or rank on the uncalibrated score and calibrate only the displayed probability.
Reason
A strictly monotone transform preserves ranking; isotonic regression is only non-decreasing and flattens regions into constants.
Boundary
Pipelines where the same score both ranks and is read as a probability.
Symptom or risk
Isotonic regression maps many distinct scores to the same value, introducing ties that can change ROC-AUC and blur the order at the top of the list.
Exceptions
With more than about a thousand calibration samples isotonic is often as good or better for probabilities, and ranking may not matter.
Trade-offs
Isotonic corrects any monotone distortion; sigmoid is smoother but assumes a particular shape.
Pitfalls
Measuring ranking on the calibration set itself, where isotonic looks better.
Example
Hypothetical: an isotonic map fitted on 1,000 calibration points can turn 5,000 distinct scores into a few dozen or a couple of hundred plateau values, and ROC-AUC on fresh data can move slightly.
Lessons and concepts
Claims and their sources
scikit-learn notes calibration generally does not affect ranking metrics such as ROC-AUC, but isotonic regression introduces ties so AUC may change; sigmoid is strictly monotonic and keeps the ranking.