LLMs, GenAI, RAG, agents, evaluation · insight-0557

Run every pairwise LLM judgment in both orders and count only consistent verdicts

Situation
An LLM judge compares old and new assistant answers, always shown in the same order, and prefers the new one 62 percent of the time.
Action
Judge each pair twice with positions swapped, count a win only when both orders agree, treat disagreements as ties, and report the inconsistency rate as a judge-quality metric.
Reason
Studies of LLM-as-a-judge document position bias, where the verdict changes with the order in which answers are presented.
Boundary
Pairwise model, prompt or fine-tune comparisons scored by an LLM.

Symptom or risk

Part of the preference comes from position rather than quality, and the release decision rests on an artefact.

Single-answer grading against a rubric avoids order effects but has other biases.

Doubles judging cost.

Randomising order once per item, which hides the bias in noise instead of measuring it.

Zheng et al. document position bias in LLM judges and propose swapping answer positions as a mitigation.
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena