Neural networks, deep learning, NLP · insight-0443
When BERT-style fine-tuning fails on some seeds, train longer with a smaller learning rate
- Evidence type
- empirical finding
- Difficulty
- hard
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Fine-tuning an encoder on a small classification set succeeds on some seeds and collapses to majority-class predictions on others.
Action
Use a smaller learning rate with warmup, Adam bias correction enabled and more training steps; run several seeds and check every run's training loss, not only the best validation score.
Reason
Mosbach et al. attribute fine-tuning instability mainly to optimization difficulties with vanishing gradients, not to forgetting or small data, and propose a simple stable baseline.
Boundary
Fine-tuning Transformer encoders on small labelled datasets.
Symptom or risk
Teams blame catastrophic forgetting or data size and abandon fine-tuning, or keep the lucky seed and ship an unstable recipe.
Exceptions
Large datasets usually fine-tune stably with standard recipes.
Trade-offs
Longer training costs compute; restarting failed seeds is a fragile alternative.
Pitfalls
Reporting only the best seed; ignoring runs whose training loss never fell.
Lessons and concepts
Claims and their sources
Mosbach et al. show fine-tuning instability of BERT-style models is caused by optimization difficulties that lead to vanishing gradients rather than by catastrophic forgetting or small dataset size.
Dodge et al. observe that on small datasets many fine-tuning trials diverge part of the way through training.