Classical ML, validation, metrics, interpretation, debugging · insight-0146
Judge probability quality with log loss or Brier score, and calibration separately
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- banking, aviation
Summary: situation, action, reason, boundary
Situation
Predicted probabilities will feed expected-value calculations, pricing or risk limits, and you only report ROC-AUC.
Action
Report log loss or Brier score next to ROC-AUC, and add a reliability diagram, because a low Brier score alone does not prove calibration.
Reason
Proper scoring rules reward honest probabilities; the Brier score mixes calibration, resolution and irreducible uncertainty, so inspect calibration directly too.
Boundary
Credit risk, no-show and overbooking models, claim probabilities and any model whose outputs are used as probabilities.
Symptom or risk
ROC-AUC is unchanged by any monotone distortion of scores, so badly miscalibrated probabilities can pass review.
Exceptions
If only ranking matters (top-K review queues), ROC- or precision-at-K metrics suffice.
Trade-offs
Log loss punishes confident mistakes harshly and is sensitive to label noise; Brier score is gentler.
Pitfalls
Comparing log loss across datasets with different prevalence, where the floor differs.
Lessons and concepts
Claims and their sources
The Brier score is a strictly proper scoring rule whose value mixes calibration, resolution and uncertainty, so a lower score does not by itself show better calibration.
Strictly proper scoring rules reward reporting one's true predictive distribution.