LLMs, GenAI, RAG, agents, evaluation · insight-1586
Rank many variants with a Bradley-Terry fit on sampled pairs rather than all-pairs win counts
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You compare eight or more prompt or model variants by pairwise judging and the all-pairs table has dozens of cells with few items each.
Action
Sample pairs across items, fit a Bradley-Terry model to the verdicts, record ties explicitly, and report each variant's score with an interval from bootstrap or robust standard errors.
Reason
One strength parameter per variant summarises all comparisons, and the intervals show which ranks are actually separated.
Boundary
Pairwise human or judge comparisons across many candidates.
Symptom or risk
Raw win counts ignore opponent strength and uneven sampling, and cycles (A beats B beats C beats A) make a table hard to turn into a ranking.
Exceptions
With two or three variants a paired win-rate with an interval is enough.
Trade-offs
The model assumes one transitive strength per variant, so check the residual pairs for cycles by slice.
Pitfalls
Reading rank order from point scores when intervals overlap.
Example
Eight variants give 28 distinct pairs; each pair judged on 50 items is 1,400 judgments, so sample pairs adaptively.
Lessons and concepts
Claims and their sources
The Chatbot Arena authors rank models from pairwise votes with the Bradley-Terry model, handle a tie option, and compute confidence intervals via pivot bootstrap or sandwich standard errors.