S14 · paper simulation

Check an LLM-judge agreement claim before adopting it

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
35 minabout
0stages committed

The brief

A support-quality team (simulated) compares replies from the current assistant (reply A) with replies from a candidate assistant (reply B) on the same customer messages. A nightly check should block a release when the candidate is worse. Human reviewers do this today; a simulated teammate proposes replacing them with an LLM judge, citing one slide:

"Judge J agrees with human reviewers on 87% of 200 pairwise comparisons. We can drop human review."

All counts below are synthetic, generated for this exercise. The registry sources on LLM judges are cited in the reveals, after you have committed to an answer.

Stage 1 of 4

The slide has one number and no table. Before you look at any data you list what you need to know to interpret the 87%. You do not yet have the confusion counts.

Which questions must be answered before the 87% means anything for this release check? Select all that apply.

What would a rule that always prefers the new reply score on the same pairs, and how often do two humans agree?Were the two replies shown in a fixed order, or was the order swapped or randomised across the 200 pairs?How many parameters does judge J have, since larger judges are less biased and need no further checks on this task?Does the judge's written explanation read fluently and cite specific phrases from each reply to show it compared them?Did the judge model also write or help write one of the two replies being compared?