Neural networks, deep learning, NLP · insight-0436
Report results over several seeds before claiming a neural improvement
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A fine-tuned text classifier scores 1.2 points higher than the current one on a single run.
Action
Run at least 3 to 5 seeds for both models, report mean and spread, and accept the change only if the gap is larger than the spread. Keep seeds fixed and logged.
Reason
Dodge et al. found substantial differences in BERT fine-tuning results from seeds alone, with weight initialization and data order contributing comparably.
Boundary
Neural training and fine-tuning, especially on small datasets.
Symptom or risk
Seed-to-seed variation from initialization and data order can be as large as the claimed gain, so the change may be noise and later "regress" for no reason.
Exceptions
When a single run is the only affordable option, state the limitation and use a larger test set.
Trade-offs
Multiplies training cost; run seeds on the shortlisted candidates only.
Pitfalls
Picking the best seed of the new model and comparing it with one seed of the old model.
Lessons and concepts
Claims and their sources
Dodge et al. report that fine-tuning BERT with distinct random seeds gives substantially different results and that weight initialization and data order contribute comparably to the variance.
Bouthillier et al. recommend accounting for multiple sources of variation when comparing learning algorithms.