Neural networks, deep learning, NLP · insight-1419
Vary the initialisation seed and the data-order seed separately to see which one moves the metric
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Two runs of the same configuration differ by half a point and you do not know whether to blame initialisation, shuffling, augmentation or hardware.
Action
Run a small grid: vary only the initialisation seed, only the data-order seed, only the augmentation seed, and report the spread for each next to the improvement you are claiming.
Reason
Bouthillier and colleagues show that variance from data sampling, initialisation and hyperparameter choice can change benchmark conclusions, so each source should be estimated.
Boundary
Any claimed improvement smaller than a point or two on a neural model.
Symptom or risk
One undifferentiated seed hides the source of variance, so a tuning comparison may be decided by the noise source you did not control.
Exceptions
Large effects that clearly exceed every source's spread.
Trade-offs
Three to five extra short runs per configuration.
Pitfalls
Using one fixed validation split for every seed, which excludes data-split variance.
Lessons and concepts
Claims and their sources
Bouthillier et al. (arXiv preprint) report that variance from data sampling, parameter initialisation and hyperparameter choice markedly affects benchmark conclusions.