Neural networks, deep learning, NLP · insight-1455
Hold out annotators, not only items, when a few people produced most of the labels or texts
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A tone classifier is trained on labels from five agents, two of whom labelled 70 percent of the rows, and the test split is random.
Action
Record annotator identifiers, evaluate with annotator-grouped splits or a test set from unseen annotators, and report the gap to the random-split score.
Reason
Geva, Goldberg and Berant found on three datasets that annotator identifiers improved performance and that models often failed to generalise to annotators absent from training; they recommend separating annotators across splits.
Boundary
Crowdsourced and agent-labelled data, and datasets written by a few authors.
Symptom or risk
The model learns each annotator's style and habits; a random split puts the same people in train and test, so the score rewards that and falls on labels from new people.
Exceptions
A fixed annotator pool that will also label production data.
Trade-offs
Fewer training annotators per fold, higher variance.
Pitfalls
Not recording annotator identity; adding the identifier as a feature and shipping it.
Lessons and concepts
Claims and their sources
Geva, Goldberg and Berant report that annotator identifiers improve performance, models often fail to generalise to annotators unseen in training, and test annotators should be kept separate from training annotators.