LLMs, GenAI, RAG, agents, evaluation · insight-0561

Use pairwise judging to rank close variants and rubric grading to set pass bars

Situation
A team asks an LLM judge to score answers from 1 to 10 to decide between two similar prompt versions.
Action
Use pairwise comparison, with order swapping, to choose between close variants, and use rubric or reference-guided grading with explicit pass criteria to decide whether a version is good enough to ship.
Reason
The LLM-as-a-judge paper lists pairwise, single-answer and reference-guided grading as variants with different pros and cons; pairwise comparison separates close candidates but needs quadratically many pairs, while single-answer scores scale to many candidates but may miss subtle differences and fluctuate more. Using explicit rubric criteria to set a pass bar is a practitioner recommendation that the paper does not test.
Boundary
Prompt iteration, model selection, release gates.

Symptom or risk

Both versions average around 7, absolute scores shift between runs, and the subtle difference that matters is lost.

Tasks with exact answers need neither; check them deterministically.

Pairwise judgments grow quadratically with the number of candidates.

Treating a pairwise win rate as evidence that either version meets the quality bar.

Zheng et al. describe pairwise comparison, single-answer grading and reference-guided grading as LLM-judge variants with different pros and cons, noting that pairwise comparison scales quadratically with the number of candidates while single-answer grading may miss subtle differences and give less stable absolute scores.
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena