LLMs, GenAI, RAG, agents, evaluation · insight-1584
Check judge recall on failing answers and chance-corrected agreement, because skewed pass rates inflate raw agreement
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A judge reports 90% agreement with human reviewers and 90% of answers are acceptable.
Action
Report Cohen's kappa or Krippendorff's alpha next to raw agreement, plus the judge's recall on the failing class, and size the validation sample so the failing class has enough items.
Reason
Observed agreement must be adjusted for the agreement expected by chance, and severely skewed category distributions make raw figures misleading.
Boundary
Judge validation and human-annotator reliability studies with imbalanced labels.
Symptom or risk
On skewed labels, a judge that approves everything agrees with humans most of the time, so raw agreement certifies a useless grader.
Exceptions
Balanced labels make the gap between raw agreement and kappa small.
Trade-offs
Kappa is itself unstable on very rare classes; show per-class counts.
Pitfalls
Applying fixed "good kappa" cut-offs; thresholds depend on the stakes.
Example
Hypothetical: a judge that always says "pass" on a 90%-pass set has 90% raw agreement and kappa of 0.
Lessons and concepts
Claims and their sources
Artstein and Poesio explain that observed agreement must be adjusted for chance agreement, and discuss how skewed (prevalence) distributions make high coefficient values hard to reach.