LLMs, GenAI, RAG, agents, evaluation · insight-1581
Decompose a judge rubric into independent yes/no checks instead of one holistic score
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Applied AI
- Industries
- aviation, telecom
Summary: situation, action, reason, boundary
Situation
A judge prompt asks "rate the answer's overall quality from 1 to 10" and scores cluster in a narrow band, or flip between runs.
Action
Write three to six binary criteria, each with a one-line definition and a pass and fail example (states the refund rule, cites the clause, no unsupported claim); report each criterion's pass rate and the share of answers passing all.
Reason
Binary checks are easier for the judge to apply consistently and for humans to audit, and per-criterion agreement shows which check to fix.
Boundary
Customer-facing answers with explicit quality requirements.
Symptom or risk
One number blends correctness, tone and completeness, so changes are unexplainable and judge-human disagreement cannot be located.
Exceptions
Pure preference ranking of near-equal outputs is better done by pairwise comparison.
Trade-offs
More judge calls, and a checklist can miss qualities nobody listed.
Pitfalls
Overlapping criteria that double-count one flaw; keep each check independent.