Introduced on day 45
Position bias
A judge's tendency to prefer an answer because of the slot it appears in (first or second) rather than its content, visible as verdicts that flip when the order is swapped.
In detail
It can leave overall agreement with humans looking acceptable while moving a candidate's win rate by tens of points depending on the order. Mitigations are judging every pair in both orders, counting only consistent verdicts and treating flips as ties, or randomising order and averaging; none removes verbosity or self-preference effects.