Neural networks, deep learning, NLP · insight-1409
When you add mixup, treat the targets as soft and change the metrics accordingly
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You add mixup to an image or tabular network that overfits a few thousand labelled rows, and keep the old accuracy and threshold code.
Action
Mix inputs and targets with the same weight, train with a loss that accepts soft targets, and keep the validation loader unmixed. Compare against the old run on validation data only.
Reason
Mixup trains on convex combinations of pairs of examples and their labels, which encourages near-linear behaviour between training points and was reported to reduce memorisation of corrupted labels.
Boundary
Classification with continuous inputs where a convex combination of two inputs is plausible.
Symptom or risk
Training targets are now blends, so training accuracy and loss are not comparable with earlier runs, and pipeline code that expects hard labels may quietly take an argmax of the mixed target.
Exceptions
Categorical IDs, text tokens and structured records do not blend meaningfully without a different mixing space.
Trade-offs
Another hyperparameter, the mixing distribution, and slower convergence.
Pitfalls
Mixing before normalisation or across different domains produces inputs no real example resembles.
Lessons and concepts
Claims and their sources
Mixup trains on convex combinations of pairs of examples and their labels, and its authors report better generalisation and less memorisation of corrupted labels.