Neural networks, deep learning, NLP · insight-1411
Recalibrate after adopting focal loss for an imbalanced classifier
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
A rare-event network (for example, a synthetic no-show flag at 2 percent prevalence) is trained with focal loss and its scores are fed into a cost-based threshold.
Action
Evaluate calibration on a held-out split, then fit temperature scaling or isotonic regression on a separate calibration split and choose the threshold on the recalibrated scores.
Reason
Focal loss down-weights well-classified examples to handle imbalance; Mukhoti et al. report on general classification benchmarks (not rare-event data) that focal-loss models can be better calibrated than cross-entropy yet still benefit from temperature scaling, so the benefit for a 2 percent class is to be checked, not assumed.
Boundary
Imbalanced classification where probabilities, not only rankings, feed decisions.
Symptom or risk
Focal loss reweights examples, so its outputs are not the class frequencies the cost calculation assumes, and the chosen threshold or expected-cost estimate is off.
Exceptions
If only the rank order is used, for example top-k review queues, calibration matters less.
Trade-offs
Needs another labelled split, which is expensive when positives are rare.
Pitfalls
Calibrating on the same split used to choose the early-stopping epoch.
Lessons and concepts
Claims and their sources
Focal loss reshapes cross-entropy to down-weight well-classified examples so many easy negatives do not dominate training.
Mukhoti et al. report that models trained with focal loss are better calibrated than with cross-entropy and that adding temperature scaling gives the best calibration while preserving accuracy.