LLMs, GenAI, RAG, agents, evaluation · insight-1582
Test the judge with planted bad answers before you let it gate releases
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Judge-human agreement looks fine on a random sample, but the answers that matter are the rare fluent-but-wrong ones.
Action
Build a negative-control set: answers to a different question, truncated answers, correct-looking answers with one wrong number, and refusals. Measure how many the judge passes and require that rate to be near zero.
Reason
A control with a known-bad label measures the judge's false-pass rate directly, independent of how rare real failures are.
Boundary
Any judge used as a pass-fail gate.
Symptom or risk
Random samples contain few bad answers, so a lenient judge passes broken outputs and the gate never fires.
Exceptions
Rank-only uses (which variant is better) tolerate a lenient judge more than gates do.
Trade-offs
Controls must resemble real failures to be informative; synthetic corruptions are easy and give false comfort.
Pitfalls
Only testing obvious corruptions; add failures harvested from production.
Example
Hypothetical: 40 planted-wrong answers, judge passes 6, a 15% false-pass rate that overall agreement would hide.