Neural networks, deep learning, NLP · insight-1454
Keep per-item annotator votes and test soft labels before collapsing them to a majority
- Evidence type
- empirical finding
- Difficulty
- hard
- Roles
- Data scientist, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Three annotators label each tone or intent example, the pipeline keeps the majority label, and a third of the items were split 2 to 1.
Action
Store every vote, train one model on the vote distribution (soft labels) beside the majority-label model, evaluate both against hard labels and against the vote distribution, and use vote entropy as a review signal.
Reason
Uma et al. surveyed training on multiple judgments and found that with large datasets and many high-quality judgments per item soft labels beat aggregated labels, and otherwise combining gold and soft labels did best on hard evaluation; results depended strongly on the dataset and evaluation.
Boundary
Subjective or ambiguous text labels such as tone, intent, toxicity and priority.
Symptom or risk
Contested items are trained and scored as certain, the model is penalised for reasonable disagreement, and the uncertainty that could route cases to a human is thrown away.
Exceptions
Objective labels with one verifiable answer.
Trade-offs
Several votes per item cost annotation budget; soft-label training needs a loss that accepts distributions.
Pitfalls
Keeping votes from annotators who never saw the guidelines; evaluating only on the majority label.
Lessons and concepts
Claims and their sources
Uma et al. report that with large datasets and many high-quality judgments per item, training on soft labels outperforms training on aggregated or gold labels, and that otherwise combining gold and soft labels generally gave the best results under hard evaluation.