LLMs, GenAI, RAG, agents, evaluation · insight-1580

Give a judge a reference solution when the question has a checkable answer

Situation
An LLM judge grades answers to arithmetic, policy-lookup or fare-calculation questions without being shown the correct result.
Action
Put the verified reference answer in the judge prompt, ask for a comparison against it with the reasoning, and spot-check a sample of verdicts against human grades.
Reason
Grading against a known answer turns a hard open-ended judgment into a narrower comparison, which a judge handled more reliably than solving from scratch in the MT-Bench authors' small test.
Boundary
Items with a single verified answer or a short list of required facts.

Symptom or risk

The judge solves the problem itself, often copies the candidate's mistake, and approves fluent wrong answers.

Open-ended writing has no single reference; use a rubric instead of one reference answer.

Every item needs a reference, which is labelling work, and a wrong reference becomes a wrong verdict.

Using the judge model's own answer as the reference; it carries the same errors.

In the MT-Bench judge study, adding a reference-guided prompt cut the GPT-4 judge's rate of accepting wrong answers on 10 math questions from 70% to 15% (14 of 20 to 3 of 20 judgments, both answer orders).
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena