LLMs, GenAI, RAG, agents, evaluation · insight-0559

Avoid letting a model family judge its own outputs

Situation
A team uses the same model to generate answers and to judge them against a competitor's answers.
Action
Use a judge from a different model family, or several judges, and confirm on a human-labelled subset that the judge does not favour any candidate.
Reason
Research found LLM evaluators rate their own generations higher than equally good outputs from other models, and this self-preference is linked to recognising their own text.
Boundary
Vendor comparisons, fine-tune versus base comparisons, self-improvement loops.

Symptom or risk

The judge favours its own style, and the in-house configuration wins comparisons that humans would call a tie.

Absolute checks with objective answers, such as exact-match extraction, are not affected.

A second provider adds cost, contracts and data-handling review.

Assuming a different model version from the same family removes the bias.

Panickssery et al. found LLM evaluators rate their own outputs higher than others' outputs that humans judge equal, and self-recognition correlates with self-preference.
LLM Evaluators Recognize and Favor Their Own Generations
Zheng et al. report self-enhancement bias among LLM-judge biases.
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena