Neural networks, deep learning, NLP · insight-0474

Use label smoothing knowing it changes calibration and distillation

Situation
You turn on label smoothing for a classifier that will later serve as the teacher for a smaller distilled model.
Action
Decide per use: enable smoothing for a model used directly, train the teacher without it when distillation is planned, and recheck calibration on validation data either way.
Reason
Müller et al. found label smoothing improves calibration but makes knowledge distillation from the smoothed teacher much less effective, because it tightens same-class representations.
Boundary
Multi-class classifiers and sequence models trained with cross-entropy.

Symptom or risk

Label smoothing can improve accuracy and calibration, but a smoothed teacher transfers much less to the student, so the distilled model disappoints.

Very noisy labels may favour smoothing even for teachers; measure.

One more hyperparameter that interacts with temperature scaling.

Comparing calibration of smoothed and unsmoothed models without post-hoc calibration on both.

Müller, Kornblith and Hinton report label smoothing improves calibration but makes distillation from a teacher trained with it much less effective.
When Does Label Smoothing Help?
PyTorch CrossEntropyLoss supports a label_smoothing argument that mixes targets with a uniform distribution.
CrossEntropyLoss (PyTorch documentation)