Neural networks, deep learning, NLP · insight-0458
Prefer loss weighting to duplicating rare texts when classes are imbalanced
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
In a ticket classifier, safety-related complaints are 0.5% of messages and the model rarely predicts them.
Action
Keep the natural distribution in validation and test, use class weights in cross-entropy or a focal loss in training, choose the threshold from costs, and recalibrate probabilities if they drive decisions.
Reason
Weighting changes how much each error costs without creating copies; imbalance corrections can miscalibrate probabilities, so thresholds and calibration need held-out data.
Boundary
Text and other neural classifiers with rare but important classes.
Symptom or risk
Random oversampling repeats the same rare texts so the network memorizes them, while resampling distorts predicted probabilities used for thresholds.
Exceptions
When rare-class examples are diverse and plentiful in absolute terms, plain training plus threshold tuning may suffice.
Trade-offs
Strong weights raise recall but also false alarms; agree the cost ratio with the operations team.
Pitfalls
Oversampling before splitting, which puts copies of the same text in train and validation.
Lessons and concepts
Claims and their sources
PyTorch CrossEntropyLoss accepts per-class weights, documented as useful for unbalanced training sets.
Lin et al. propose focal loss, which down-weights well-classified examples so that many easy negatives do not dominate training.
van den Goorbergh et al. found random over- and undersampling and SMOTE produced strongly miscalibrated risks without improving discrimination in their study.