Skip to content
ML Academy

Introduced on day 45

LLM-as-judge

Using a language model to grade or compare other models' answers against a rubric, which scales evaluation but adds the judge's own biases.

In detail

Known effects include preference for the answer in a given position, for longer answers and for answers from the judge's own model family. A judge must be validated against human labels on the target task, run in both orders for pairwise comparisons, and re-validated when the judge model or prompt changes. Programmatic checks should be used first wherever the criterion can be verified by code.