LLMs, GenAI, RAG, agents, evaluation · insight-1596
Pilot an annotation guide on 30 items and treat low agreement as a spec defect
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Two annotators label LLM outputs for "helpfulness" and agree only slightly better than chance.
Action
Before scaling, have two or more annotators label 30 items, list each disagreement and what rule would settle it, add decision rules and worked edge cases to the guide, then re-pilot until agreement is stable; adjudicate remaining disagreements with a third reader and keep both original labels.
Reason
Disagreement between careful annotators usually points to an under-specified criterion, and fixing the guide is cheaper than relabelling thousands of items.
Boundary
Rubric-based human evaluation of generated text.
Symptom or risk
Teams blame annotators or average the labels, which bakes an ambiguous definition into the metric and into any judge calibrated against it.
Exceptions
Genuinely subjective preference tasks will keep some disagreement; report it instead of forcing consensus.
Trade-offs
One or two extra rounds before labelling at scale.
Pitfalls
Pilot items that are all easy; include borderline cases.