Neural networks, deep learning, NLP · insight-1414
Do not quote one expected-calibration-error number without its binning and a reliability diagram
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A report compares two networks by a single ECE value computed with ten equal-width bins on a test set of 800 rows.
Action
Report ECE with the bin count and scheme, repeat it with adaptive (equal-mass) bins, add a reliability diagram with bin counts, and give a bootstrap interval for the difference.
Reason
Nixon and colleagues showed that expected calibration error has known flaws and that measurement choices can change which recalibration method ranks best.
Boundary
Comparing classifiers or post-hoc calibrators with probabilities used downstream.
Symptom or risk
The value depends on the binning, sparse bins are noisy, and the ranking of the two models, or of the recalibration methods applied, can flip with the measurement choice.
Exceptions
When probabilities are used only for ranking, report ranking metrics instead.
Trade-offs
More numbers to explain to a business reader; keep the diagram and drop the extra columns.
Pitfalls
Treating a low ECE as proof of calibration in every segment; check by route, month or class.
Lessons and concepts
Claims and their sources
Nixon et al. (arXiv preprint) state that expected calibration error has known flaws and show that measurement design choices can change which recalibration method ranks best.