S32 · paper simulation
A language-model judge that agrees with humans 85% of the time
People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.
- 5stages
- 35 minabout
- 0stages committed
The brief
Your team wants to replace weekly human review of assistant answers with an automatic judge. A colleague forwards a preprint. The paper and its figures are fictional, written for this exercise. Its abstract (paraphrased):
"JudgeLite, a small judge model, agrees with human annotators on 85% of 200 pairwise comparisons of answers from two assistants, which makes it a drop-in replacement for human evaluation in pairwise testing."
Facts from the methods section (fictional): human labels are the majority of three annotators; the judge sees each pair once, with the answers in a fixed order; one of the two assistants is built on the same model family as the judge. The 200 pairs, by label, are in the table (rows: human majority; columns: judge).
| Judge picks A | Judge picks B | |
|---|---|---|
| Humans pick A | 130 | 20 |
| Humans pick B | 10 | 40 |
In the 120 pairs where one answer is at least 30% longer than the other, the judge picked the longer answer 94 times (78.3%) and humans picked it 73 times (60.8%).
Stage 1 of 5
You start with a checklist for any claim that a judge replaces humans.
Which checks belong in the protocol? Select all that apply.