Neural networks, deep learning, NLP · insight-1453
Count selection data as part of the budget when you only own a few dozen labels
- Evidence type
- empirical finding
- Difficulty
- hard
- Roles
- Data scientist, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A few-shot text classifier is reported at a score tuned on prompts and learning rates using a large extra validation set, though production will have only 60 labels.
Action
Select prompts and hyperparameters using only the labels you will really hold, with repeated cross-validation, keep the candidate list short and treat the choice as close to a coin flip, fix the recipe before looking at the final test, and report the variance across choices.
Reason
Perez, Kiela and Cho report in a preprint that with only the few examples, cross-validation and minimum description length chose language-model prompts and hyperparameters only slightly better than random and far worse than held-out selection.
Boundary
Few-shot prompting and tuning of small encoders.
Symptom or risk
The reported score assumes a selection set you will not have; with only the few labels, the chosen setup may be little better than a random pick.
Exceptions
When labels are cheap, collect more and the problem recedes.
Trade-offs
Fewer options explored; wider honest error bars.
Pitfalls
Reusing the final test set for choices; counting a borrowed dev set as free.
Lessons and concepts
Claims and their sources
Perez, Kiela and Cho (arXiv preprint) report that with only a few labelled examples, cross-validation and minimum description length select language-model prompts and hyperparameters only slightly better than random and far worse than selection with held-out data.