Neural networks, deep learning, NLP · insight-0461

Expect large networks to memorize noisy training labels and plan for it

Situation
Training labels come from agents' quick tags and you train a large model for many epochs until training accuracy nears 100%.
Action
Use early stopping on clean-ish validation data, inspect the highest-loss training examples for mislabels, and relabel or down-weight confirmed noise before training longer.
Reason
Zhang et al. showed standard networks can fit completely random labels, so a perfect training fit says nothing about label quality.
Boundary
Neural models trained on weak, crowd or operational labels.

Symptom or risk

Late epochs fit the noise, so validation performance peaks and then declines while training metrics keep improving.

Small, strongly regularized models memorize less, but they still absorb systematic label bias.

Relabeling costs time; down-weighting can also suppress hard but correct examples.

Removing every high-loss example, which deletes the hardest true cases.

Zhang et al. show convolutional networks trained with SGD easily fit a random labelling of the training data, and that this is qualitatively unaffected by explicit regularization.
Understanding deep learning requires rethinking generalization