Neural networks, deep learning, NLP · insight-1434
Fit a learning curve over training-set size before buying more labels or a larger model
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A team plans to triple labelling spend because validation error is still above target, without knowing how error has moved with data so far.
Action
Train on 10, 20, 40 and 80 percent of the data over three seeds, plot validation error against size on log axes, and extrapolate cautiously to see whether the target is reachable at all.
Reason
Across several domains generalisation error was reported to fall as a power law in training-set size, with architecture changes shifting the curve; a few points reveal the slope and a likely floor.
Boundary
Supervised networks where labels cost money, and for deciding between data and modelling work.
Symptom or risk
The budget may buy almost nothing if the curve is flat, or may be unnecessary if a cheaper change shifts it.
Exceptions
Tiny datasets, where the first points are too noisy to fit; report them without a line.
Trade-offs
Several extra short runs; reuse the same hyperparameters per fraction, retuned if the scale changes a lot.
Pitfalls
Extrapolating a power law beyond the sampled range as if it were certain.
Lessons and concepts
Claims and their sources
Hestness et al. (arXiv preprint) report that generalisation error falls as a power law in training-set size across machine translation, language modelling, image classification and speech, with architecture changes shifting the curve.