LLMs, GenAI, RAG, agents, evaluation · insight-0560

Validate an LLM judge against human labels before trusting its scores

Situation
A team plans to replace human review of complaint replies with an LLM judge to speed up evaluation.
Action
Have two humans label a stratified subset (for example 100-200 items) per criterion, measure judge-human agreement next to human-human agreement, and keep humans on criteria where the judge falls short.
Reason
Published agreement figures come from specific benchmarks and judges; agreement on your criteria and data must be measured, not assumed.
Boundary
Any automated LLM-as-judge evaluation that informs decisions.

Symptom or risk

The judge agrees with humans on easy cases but drifts on the criteria that matter, such as policy correctness, and nobody measures it.

Criteria with deterministic checks need no judge at all.

Human labelling is slow, but it is a one-time calibration plus periodic re-checks.

Validating the judge on the same prompt variants you will later optimise against.

Zheng et al. report that strong LLM judges reached over 80 percent agreement with human preferences on their benchmarks, similar to agreement between humans.
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena