project-042 · LLM and GenAI · cross-industry

LLM-as-judge reliability study: agreement with human pairwise labels, position bias and length bias

An evaluation team wants to replace part of its human pairwise review with an LLM judge. Is the judge reliable enough to rank model versions: how often does it agree with expert human preferences, how often does its verdict flip when the two answers swap places, and does it prefer longer answers regardless of quality?

Dataset: MT-Bench human judgments (lmsys/mt_bench_human_judgments), human split, plus the recorded gpt4_pair split
Licence: CC BY 4.0 (Hugging Face dataset card)
Reference run: executed
Lessons: day 45 day 51

Data
MT-Bench human judgments (lmsys/mt_bench_human_judgments), human split, plus the recorded gpt4_pair split (public) · Human split 3,355 rows (80 questions, 6 models, turns 1 and 2, 65 judge ids); gpt4_pair 2,400 rows
Details on data
Baseline
Two references: (1) human-human agreement on multiply-judged items, the ceiling a judge cannot be expected to beat;
Details on baseline
Validation
Sample about 300 human-judged items stratified by turn and model pair, with a fixed seed; keep 100 for prompt development and 200 for the frozen evaluation.
Details on validation
Metric
Agreement with humans with and without ties, Cohen's kappa, order-flip rate, consistency-adjusted agreement, length-bias regression coefficient, self-consistency across…
Details on metric

Success criterion

Use the judge for routine comparisons only if, on the sampled items, judge-human agreement (ties excluded) is within the human-human agreement interval measured on the same data, the order-flip rate is below 10% after order balancing, and the length-bias analysis shows no significant preference for the longer answer conditional on the human label (targets set in advance). Otherwise keep humans in the loop and report which items the judge can handle.

Context

Day 45 introduces pairwise and model-based evaluation; Day 51 uses judges for RAG metrics. Judges are evaluation instruments and need the same validation as any measurement. The MT-Bench human judgments give expert pairwise labels on 80 multi-turn questions, so the judge can be tested against people instead of being trusted.

Data

Dataset: MT-Bench human judgments (lmsys/mt_bench_human_judgments), human split, plus the recorded gpt4_pair split (public)
Licence: CC BY 4.0 (Hugging Face dataset card)
Size: Human split 3,355 rows (80 questions, 6 models, turns 1 and 2, 65 judge ids); gpt4_pair 2,400 rows
Verification: download verified

Access

Hugging Face parquet files (no login, not gated). The human split has 3,355 pairwise judgments; the gpt4_pair split has 2,400 recorded GPT-4 judge decisions from 2023. Cache the parquet files with hashes.

Grain

One row per (question, model pair, turn, human judge) with the winner label; conversations hold both answers.

Column	Meaning
question_id / turn	MT-Bench question and conversation turn (1 or 2)
model_a / model_b	The two models whose answers are compared (2023-era models)
winner	model_a, model_b or tie (human split counts 1,293 / 1,282 / 780)
judge	Anonymised human judge id
conversation_a / conversation_b	Full multi-turn conversations shown to the judge

EDA and data-quality rules

Count items judged by more than one human (same question, model pair, turn); this determines whether a human-human agreement ceiling can be estimated and with what precision.
Check the label balance by position (model_a vs model_b wins) in the human data before blaming the LLM judge for position effects.
Compute answer lengths per side and the length difference distribution; flag truncated or empty answers.

Baseline

Two references: (1) human-human agreement on multiply-judged items, the ceiling a judge cannot be expected to beat; (2) a trivial "prefer the longer answer" rule, which a judge must beat to show it measures more than length.

Features

Judge protocol: a fixed rubric prompt, the question, both answers, and a required verdict of A, B or tie in a validated format. Each sampled item is judged twice with the answer order swapped; the balanced verdict counts a win only when both orders agree, otherwise a tie. Record judge model id, version, date, temperature and the prompt version.

Candidate methods

Single-order judging
The cheap default; included to measure how much position bias it hides.
Order-balanced judging (both orders, disagreement becomes tie)
Removes the first-order effect of position bias at twice the cost.
Order-balanced judging with a written rationale before the verdict
Tests whether asking for evidence first changes agreement; costs more output tokens.

Validation design

Sample about 300 human-judged items stratified by turn and model pair, with a fixed seed; keep 100 for prompt development and 200 for the frozen evaluation. Agreement statistics use bootstrap intervals over questions, because items from the same question are correlated.

Leakage checks

MT-Bench and its judgments are public and may be in a judge model's training data; a judge could reproduce published verdicts. Record the judge's stated training cut-off and treat high agreement with caution; a transfer check on a few new questions with your own human labels addresses this.

Bias and representativeness

Self-preference: if the judge belongs to the same family as one of the compared models, report agreement separately for those pairs. Human labels come from graduate-level annotators (dataset card), not end users.

Robustness checks

Rerun 50 items twice to measure judge self-consistency; vary the rubric wording once to see whether agreement depends on phrasing.

Technical metrics

Agreement with humans with and without ties, Cohen's kappa, order-flip rate, consistency-adjusted agreement, length-bias regression coefficient, self-consistency across reruns; all with bootstrap intervals.

Business interpretation

Share of review volume the judge can take over at the agreed quality, cost per 1,000 judgments under a labelled hypothetical price, and the residual human review needed for disagreement cases.

Error slices

Turn 1 vs turn 2 (follow-up questions are harder to judge).
Question category and model pair (close pairs vs lopsided pairs).
Length difference quantiles.

Implementation steps

Load and profile the human split; estimate human-human agreement where overlap exists.
Without a model: analyse the recorded gpt4_pair split against human labels (agreement, length bias). Check whether it records both answer orders before attempting any position-bias analysis, and label it as recorded 2023 judgments of one model.
With a configured judge model: implement the protocol with validated verdict parsing; develop on 100 items.
Freeze the prompt; judge 200 evaluation items in both orders; compute all metrics with intervals.
Write the reliability report and a usage policy (where the judge can replace people and where it cannot).

Reproduction

Learner-authored; for example uv run python -m judge.profile, uv run python -m judge.recorded --split gpt4_pair and, with a model configured, uv run python -m judge.run --items eval --orders both. Fixed sampling seed; every judge call is logged with model version and prompt hash so the run can be audited.

Limitations

The compared answers come from 2023-era models; judge reliability on today's stronger answers may differ.
Agreement with these experts does not establish agreement with your product's users or domain experts.
Judge behaviour is version-specific; a provider update requires re-validation.

Production and monitoring

Treat the judge as a versioned instrument: pin model and prompt, keep a calibration set of human-labelled items, and re-run it on every judge change. Route judge disagreements across orders and low-confidence items to people. Track agreement drift monthly on fresh human labels.

Interview questions

How would you show a sceptical stakeholder that an LLM judge measures quality and not answer length?
Why compare judge-human agreement with human-human agreement rather than with 100%?
What changes in your evaluation if the judge model is from the same family as the model being evaluated?

Postmortem and alternatives

If the judge agrees with humans on lopsided pairs but not on close ones, it is useful for regression detection and not for fine ranking. Discuss whether a cheaper rubric-based single-answer grading would meet the same need.

Reference implementation

Status: executed — outputs below are from the recorded run

cd python && uv run python -m reference.r18_llm_judge_reliability.run

r18 results - judge-reliability study on recorded synthetic judge outputs

Synthetic data: no real judge and no real annotator was evaluated. Every verdict is a seeded draw from a generative model with injected biases (simulator.py, recorded in fixtures/judge_records_v1.json, seed 20241013). The results show what the analysis and the decision rule do on judges whose bias is known. They are not reliability figures for any model and must not be quoted as such. Numbers are copied from outputs/metrics.json and outputs/results_tables.md, written by cd python && uv run python -m reference.r18_llm_judge_reliability.run.

Item	Value
Date of run	2026-10-12
Seeds	20241013 (generator, bootstrap, gate simulation seed + 7)
Runtime	about 6 s full run, about 1 s `--smoke` (Python 3.12.13, numpy 2.5.3); the 18 tests pass in about 3 s
Data	300 items, 80 questions; dev 100 items (24 questions), evaluation 200 items (55 questions); 93 evaluation items have a second human; 50 evaluation items re-judged
Human labels (evaluation, human 1)	a 94, b 90, tie 16; first-position win share among decisive 51.1% (Wilson 43.9% to 58.2%): no position imbalance in the human data
Intervals	percentile bootstrap over questions, 1000 draws; Wilson for simple proportions

Findings

Reference points. Human 1 versus human 2 on the 93 doubly labelled items: 71.0% agreement with ties included (62.4% to 79.4%), 84.2% with ties excluded (77.1% to 91.1%, 82% of items retained), kappa 0.503 (0.373 to 0.631). The floor "prefer the longer answer" reaches 53.5% (45.9% to 61.0%) and kappa 0.180 (0.055 to 0.303), so length alone carries signal, which is exactly why length bias has to be tested against it.
The clean judge sits at or above the human-human ceiling, by construction. judge-clean with the order-balanced protocol: 73.0% ties included, 91.5% ties excluded (87.2% to 95.8%, 76% retained), kappa 0.557 (0.453 to 0.660). The generator gives the judge less noise than a human, so "within the human-human interval" is built in; the interesting comparisons are among judges. judge-noisy falls clearly below it: kappa 0.339 (0.230 to 0.451), 78.6% ties excluded.
Position bias is large and visible in three independent places, and a high ties-excluded agreement hides it. judge-position: order-flip rate 41.5% (34.5% to 48.5%) against 8.5% (5.0% to 12.6%) for the clean judge; the first shown answer wins 65.3% (62.3% to 68.4%) of its decisive calls; the position intercept of the regression is 1.42 (0.83 to 5.71, interval wide because some resamples are close to separation). Order balancing removes the bias and changes the headline: ties-excluded agreement rises to 96.4% while coverage falls from 82% to 56%, ties-included agreement falls from 74.0% to 58.5% and kappa from 0.558 to 0.405. A report that quotes only ties-excluded agreement would rank this judge best.
Order balancing fixes position bias and nothing else. In the gate simulation a single-order judge with the candidate first ships a candidate with no true advantage in 100% of null scenarios (160/160) for judge-position; order-balanced it ships 3.1% (5/160), close to the single human's 8.1% (13/160, partly the humans' small length preference, see below). For judge-length the false-ship rate at the null is 50.6% (81/160) under both protocols, and for judge-selfpref with a candidate from the judge's family it is 93.8% single-order and 95.6% balanced.
Length preference is recovered, but the raw coefficient needs a human reference. judge-length: 2.01 logits per SD of log length ratio (1.35 to 3.45), against 0.23 (-0.50 to 0.85) for the clean judge and 0.43 (-0.21 to 1.31) for human 2 on the same kind of regression: an unbiased rater still shows a positive coefficient because the label is noisy and length tracks quality. Judge minus human-2 reference on the 93 doubly labelled items: 2.42 (0.72 to 6.48), flagged. The same test does not flag judge-mixed (1.23, interval -0.002 to 5.06) although its raw coefficient is 1.20 (0.30 to 2.73) and a length weight of 0.4 was injected: with 93 items the reference-based test has low power.
Self-preference barely moves the headline and shows only in the conditional analysis. judge-selfpref single-order agreement is 70.0% (63.7% to 77.2%) against 73.5% for the clean judge, which a reader would not notice. The family coefficient is 1.64 (0.98 to 2.92) against -0.49 (-1.48 to 0.46) for the clean judge, and the gate simulation shows the consequence (finding 4).
Stated confidence is overconfident and a dev-fitted recalibration repairs most of it. judge-clean: mean stated confidence 0.877 against 0.735 agreement; ECE 0.142 (0.086 to 0.201) raw and 0.063 (0.028 to 0.124) after an isotonic map fitted on the 100 dev items. Of the 128 items stated at 0.9 to 1.0, 88.3% agree with the human; of the 18 stated at 0.5 to 0.6, 33.3% do. For judge-noisy ECE raw is 0.265. Keeping only order-balanced verdicts with both orders at or above 0.9 confidence gives 88.4% agreement (ties included) on 121 items (60% of the set) for the clean judge.
Flip rate does not see consistent noise. judge-noisy has a flip rate of 8.0%, as low as the clean judge's, because its error is shared across presentation orders; it fails on agreement and calibration, not on order. Its agreement on turn 2 is 69.4% against 86.6% on turn 1 and 70.0% on close pairs against 87.8% on lopsided pairs (ties excluded, n = 87 to 113 per slice), the pattern the project record's postmortem describes. The clean judge shows no such gap (91.4% / 91.7% by turn).
Order balancing did not change the system ranking here. Kendall tau of the six systems' win shares against human 1 is 1.00 for the clean, position, length and mixed judges under both protocols; 0.73 (single) and 0.87 (balanced) for the noisy judge; 0.87 for both protocols for the self-preferring judge. The position bias is symmetric across systems (order is random), so it distorts a candidate-first comparison (finding 4), not a round-robin ranking.

Decision rule outcome (tiers):

Judge	R1	R2 flip < 10%	R3 length	R4 family	R5 ECE <= 0.08	R6 rerun >= 0.80	Tier
judge-clean	pass	pass	pass	pass	pass	pass	gate
judge-noisy	FAIL	pass	pass	pass	FAIL	pass	do_not_use
judge-position	pass	FAIL	pass	pass	FAIL	pass	screen_and_triage_only
judge-length	pass	pass	FAIL	pass	FAIL	pass	screen_and_triage_only
judge-selfpref	pass	pass	pass	FAIL	pass	pass	screen_and_triage_only
judge-mixed	pass	FAIL	pass	pass	pass	pass	screen_and_triage_only

Only the clean judge may gate a release. The R5 failures at 0.081 (position, length) are within the interval around the 0.08 threshold (for example 0.04 to 0.13 for judge-position); none of those judges' tiers hinges on R5. The clean judge's flip rate passes R2 as a point estimate (8.5%), but its interval (5.0% to 12.6%) does not lie below 10%. judge-mixed is missed by R3 (finding 5) and caught by R2 only because it also has a position component.

The analysis recovers the injected structure. Estimated position intercepts: 1.42 and 0.84 for the injected bonuses 0.40 and 0.20 (their intervals exclude 0), and 0.17, 0.01, 0.26, 0.08 (intervals including 0) for the four judges without one; estimated length coefficients and family coefficients rank in the same order as the injected weights (the recovery table below).

Project success criterion, status: judge-human agreement within the human-human interval: met by the clean judge (by construction), not met by the noisy judge; order-flip rate below 10% after balancing: met as a point estimate by the clean, noisy, length and self-preferring judges, with the clean judge's upper bound at 12.6%; no preference for the longer answer conditional on the human label: met only by judges without an injected length weight, and not detected for judge-mixed. The simulated "judge may gate releases" decision therefore reduces to one judge of six. These statements are about the simulation, not about any real judge.

What this does not prove

It does not measure any real judge, model, annotator or benchmark. All verdicts and labels are generated; no MT-Bench data was used.
Recovering the injected biases is expected: they are additive on a latent scale, the noise is Gaussian, and the regression has the same structure. Real judges have non-additive, item-dependent biases (style, formatting, answer correctness interacting with length), which this analysis would not necessarily separate.
"Judge within the human-human interval" is partly a design choice: the clean judge's noise is lower than the humans'. A real judge may be worse than a second human, and real humans disagree in structured ways (annotator effects) that are modelled here as independent noise.
The human labels carry a small length preference (0.10) and are otherwise unbiased, so the gate simulation's human false-ship rate (8.1% at the null) mixes sampling variation with that preference; no real human panel is this clean.
200 evaluation items from 55 questions give wide intervals (for example kappa 0.45 to 0.66 for the clean judge); the differences between the clean judge and the mixed or self-preferring judges on the headline agreement numbers are within those intervals.
Thresholds R1 to R6 are author conventions chosen before the run, not derived; R5 sits near the noise floor of a five-bin ECE on 200 items. The 10% flip threshold comes from the project record.
Confidence is simulated as a probability-like quantity. Verbalised confidence from real models is often uninformative or saturated, and many judges provide none.
The gate simulation uses the same generator as the recorded data, so it shows how the protocols behave under this model, not how often a real gate errs; it also assumes one candidate-first ordering for the single-order protocol.
The dev split is used only to fit the calibration map; the project's "100 items for prompt development" and rubric-variation / rationale-first candidates were not simulated, and the rationale-before-verdict variant has no honest synthetic analogue.

Reproduce

cd python && uv run python -m reference.r18_llm_judge_reliability.run
cd python && uv run pytest tests/test_r18_llm_judge_reliability.py

Tables (copied from outputs/results_tables.md)

Reference points (evaluation split)

Reference	Agreement, ties included	Agreement, ties excluded (coverage)	Cohen kappa (3 class)
human 1 vs human 2, 93 doubly labelled items	71.0% [62.4, 79.4]	84.2% [77.1, 91.1] (82%)	0.503 [0.373, 0.631]
rule: prefer the longer answer (tie within 5%), vs human 1	53.5% [45.9, 61.0]	62.4% [54.3, 69.9] (85%)	0.180 [0.055, 0.303]

Agreement of each judge with human 1 (evaluation split, cluster bootstrap over questions)

Judge	Single order AB: ties incl	ties excl (coverage)	kappa	Order-balanced: ties incl	ties excl (coverage)	kappa
judge-clean	73.5% [66.7, 80.6]	89.4% [84.2, 94.1] (80%)	0.550 [0.445, 0.659]	73.0% [65.8, 80.0]	91.5% [87.2, 95.8] (76%)	0.557 [0.453, 0.660]
judge-noisy	62.0% [54.8, 69.4]	78.0% [70.9, 85.0] (80%)	0.353 [0.243, 0.465]	60.5% [53.3, 67.9]	78.6% [71.2, 85.9] (77%)	0.339 [0.230, 0.451]
judge-position	74.0% [68.0, 80.1]	86.0% [81.8, 90.3] (82%)	0.558 [0.470, 0.649]	58.5% [51.2, 65.2]	96.4% [92.6, 99.2] (56%)	0.405 [0.319, 0.486]
judge-length	73.0% [67.3, 79.2]	88.7% [83.3, 94.0] (80%)	0.548 [0.457, 0.645]	71.5% [64.8, 78.3]	90.1% [85.3, 94.9] (76%)	0.534 [0.438, 0.634]
judge-selfpref	70.0% [63.7, 77.2]	84.5% [79.0, 89.8] (80%)	0.492 [0.389, 0.600]	67.5% [61.2, 74.5]	83.9% [78.5, 89.3] (78%)	0.462 [0.365, 0.567]
judge-mixed	72.0% [65.3, 78.9]	90.4% [85.7, 94.7] (78%)	0.531 [0.429, 0.633]	67.0% [60.6, 73.0]	91.9% [87.2, 96.1] (68%)	0.491 [0.402, 0.570]

Position bias

Judge	Order-flip rate	Decisive flips (A vs B)	First-position win share (decisive calls, both orders)	Position intercept b0 (logit, order AB)	Rerun agreement (50 items)
judge-clean	8.5% [5.0, 12.6]	0.0% [0.0, 0.0]	49.6% [48.6, 50.6]	0.173 [-0.373, 0.966]	94% (kappa 0.90)
judge-noisy	8.0% [4.2, 12.6]	0.0% [0.0, 0.0]	50.0% [49.2, 50.9]	0.007 [-0.382, 0.406]	92% (kappa 0.87)
judge-position	41.5% [34.5, 48.5]	10.0% [6.5, 13.7]	65.3% [62.3, 68.4]	1.418 [0.834, 5.714]	90% (kappa 0.83)
judge-length	6.5% [3.6, 9.8]	0.0% [0.0, 0.0]	50.1% [49.2, 51.1]	0.256 [-0.308, 0.960]	98% (kappa 0.97)
judge-selfpref	7.0% [3.8, 10.1]	0.0% [0.0, 0.0]	49.4% [48.3, 50.5]	0.077 [-0.366, 0.497]	94% (kappa 0.90)
judge-mixed	24.5% [19.1, 29.7]	0.5% [0.0, 1.6]	57.4% [55.6, 59.4]	0.837 [0.279, 2.446]	82% (kappa 0.70)

Length and family preference (logit of the judge picking answer a, ties dropped, controlling for the human label)

Judge	Length coefficient per SD of log length ratio	Judge minus human-2 reference on the doubly labelled items	Family coefficient (answer from the judge's family is answer a)
judge-clean	0.229 [-0.495, 0.853]	-0.167 [-1.248, 0.484]	-0.487 [-1.482, 0.463]
judge-noisy	0.272 [-0.102, 0.796]	-0.340 [-1.285, 0.369]	-0.446 [-1.007, 0.125]
judge-position	0.186 [-0.456, 0.763]	-0.406 [-1.524, 0.448]	-0.281 [-1.161, 0.429]
judge-length	2.006 [1.345, 3.449]	2.418 [0.717, 6.481]	0.148 [-0.602, 1.046]
judge-selfpref	0.155 [-0.486, 0.782]	-0.489 [-2.104, 0.496]	1.643 [0.979, 2.915]
judge-mixed	1.203 [0.302, 2.732]	1.232 [-0.002, 5.061]	-0.959 [-2.703, 0.170]

Human-2 reference on the same rows: length coefficient 0.429 [-0.209, 1.309], family coefficient -0.609 [-2.097, 0.610].

Calibration of stated confidence (single-order verdict vs human 1; isotonic map fitted on the dev split)

Judge	Mean stated confidence	Agreement with human	ECE raw	ECE after dev recalibration	Brier raw	Brier recalibrated
judge-clean	0.877	0.735	0.142 [0.086, 0.201]	0.063 [0.028, 0.124]	0.174	0.158
judge-noisy	0.885	0.620	0.265 [0.197, 0.332]	0.090 [0.041, 0.149]	0.260	0.199
judge-position	0.875	0.740	0.140 [0.101, 0.194]	0.081 [0.038, 0.131]	0.187	0.172
judge-length	0.883	0.730	0.153 [0.103, 0.206]	0.081 [0.041, 0.131]	0.168	0.148
judge-selfpref	0.885	0.700	0.185 [0.130, 0.248]	0.024 [0.013, 0.091]	0.207	0.164
judge-mixed	0.891	0.720	0.179 [0.121, 0.241]	0.035 [0.014, 0.104]	0.197	0.167

Reliability bins, judge-clean:

Stated confidence bin	n	Mean stated	Agreement with human
0.5-0.6	18	0.553	0.333
0.6-0.7	21	0.654	0.429
0.7-0.8	16	0.747	0.438
0.8-0.9	17	0.856	0.706
0.9-1.0	128	0.978	0.883

Reliability bins, judge-mixed:

Stated confidence bin	n	Mean stated	Agreement with human
0.5-0.6	13	0.549	0.615
0.6-0.7	13	0.662	0.308
0.7-0.8	21	0.748	0.19
0.8-0.9	19	0.844	0.632
0.9-1.0	134	0.975	0.866

Selective use: order-balanced verdicts kept only above a stated-confidence threshold (agreement with human 1, ties included)

Judge	min confidence 0.0 (both orders agree)	0.7	0.8	0.9
judge-clean	0.77 on 183 (92%)	0.838 on 148 (74%)	0.862 on 138 (69%)	0.884 on 121 (60%)
judge-noisy	0.658 on 184 (92%)	0.722 on 151 (76%)	0.755 on 143 (72%)	0.794 on 126 (63%)
judge-position	0.915 on 117 (58%)	0.931 on 102 (51%)	0.923 on 91 (46%)	0.974 on 77 (38%)
judge-length	0.759 on 187 (94%)	0.831 on 154 (77%)	0.899 on 138 (69%)	0.913 on 126 (63%)
judge-selfpref	0.715 on 186 (93%)	0.826 on 155 (78%)	0.855 on 138 (69%)	0.878 on 123 (62%)
judge-mixed	0.828 on 151 (76%)	0.873 on 134 (67%)	0.902 on 122 (61%)	0.918 on 110 (55%)

Slices (order-balanced verdict, ties excluded; point estimates)

Judge	turn 1	turn 2	lopsided pairs	close pairs
judge-clean	0.914 (n=113)	0.917 (n=87)	0.926 (n=95)	0.903 (n=105)
judge-noisy	0.866 (n=113)	0.694 (n=87)	0.878 (n=95)	0.700 (n=105)
judge-position	0.947 (n=113)	0.982 (n=87)	0.969 (n=95)	0.957 (n=105)
judge-length	0.889 (n=113)	0.915 (n=87)	0.934 (n=95)	0.868 (n=105)
judge-selfpref	0.819 (n=113)	0.861 (n=87)	0.852 (n=95)	0.824 (n=105)
judge-mixed	0.892 (n=113)	0.952 (n=87)	0.957 (n=95)	0.881 (n=105)

Does order balancing change which system wins? (Kendall tau of system win shares against human 1)

Judge	tau single order	tau order-balanced
judge-clean	1.0	1.0
judge-noisy	0.7333	0.8667
judge-position	1.0	1.0
judge-length	1.0	1.0
judge-selfpref	0.8667	0.8667
judge-mixed	1.0	1.0

Release-gating decision rule

Judge	R1	R2	R3	R4	R5	R6	Tier
judge-clean	pass	pass	pass	pass	pass	pass	gate
judge-noisy	FAIL	pass	pass	pass	FAIL	pass	do_not_use
judge-position	pass	FAIL	pass	pass	FAIL	pass	screen_and_triage_only
judge-length	pass	pass	FAIL	pass	FAIL	pass	screen_and_triage_only
judge-selfpref	pass	pass	pass	FAIL	pass	pass	screen_and_triage_only
judge-mixed	pass	FAIL	pass	pass	pass	pass	screen_and_triage_only

Gate simulation (480 scenarios, fresh seed): how often the gate ships

Rater and protocol	False ship, true gap <= 0	False ship, null gap 0	False ship, padded (verbose) candidate	False ship, candidate from judge's family	Ships when true gap is +0.3
human	13/320 = 4.1%	13/160 = 8.1%	10/160 = 6.2%	8/160 = 5.0%	160/160 = 100.0%
judge-clean	single_candidate_first	2/320 = 0.6%	2/160 = 1.2%	0/160 = 0.0%	1/160 = 0.6%
judge-clean	order_balanced	5/320 = 1.6%	5/160 = 3.1%	2/160 = 1.2%	1/160 = 0.6%
judge-noisy	single_candidate_first	3/320 = 0.9%	3/160 = 1.9%	1/160 = 0.6%	1/160 = 0.6%
judge-noisy	order_balanced	2/320 = 0.6%	2/160 = 1.2%	1/160 = 0.6%	1/160 = 0.6%
judge-position	single_candidate_first	208/320 = 65.0%	160/160 = 100.0%	110/160 = 68.8%	104/160 = 65.0%
judge-position	order_balanced	5/320 = 1.6%	5/160 = 3.1%	4/160 = 2.5%	1/160 = 0.6%
judge-length	single_candidate_first	81/320 = 25.3%	81/160 = 50.6%	77/160 = 48.1%	40/160 = 25.0%
judge-length	order_balanced	81/320 = 25.3%	81/160 = 50.6%	77/160 = 48.1%	40/160 = 25.0%
judge-selfpref	single_candidate_first	152/320 = 47.5%	82/160 = 51.2%	76/160 = 47.5%	150/160 = 93.8%
judge-selfpref	order_balanced	156/320 = 48.8%	83/160 = 51.9%	79/160 = 49.4%	153/160 = 95.6%
judge-mixed	single_candidate_first	142/320 = 44.4%	140/160 = 87.5%	82/160 = 51.2%	67/160 = 41.9%
judge-mixed	order_balanced	55/320 = 17.2%	55/160 = 34.4%	52/160 = 32.5%	27/160 = 16.9%

Recovery of the injected bias (the analysis sees only recorded verdicts)

Judge	injected position bonus	estimated b0	injected length weight	estimated length coef	injected family weight	estimated family coef	order-flip rate
judge-clean	0.0	0.173 [-0.373, 0.966]	0.0	0.229 [-0.495, 0.853]	0.0	-0.487 [-1.482, 0.463]	8.5%
judge-noisy	0.0	0.007 [-0.382, 0.406]	0.0	0.272 [-0.102, 0.796]	0.0	-0.446 [-1.007, 0.125]	8.0%
judge-position	0.4	1.418 [0.834, 5.714]	0.0	0.186 [-0.456, 0.763]	0.0	-0.281 [-1.161, 0.429]	41.5%
judge-length	0.0	0.256 [-0.308, 0.960]	0.8	2.006 [1.345, 3.449]	0.0	0.148 [-0.602, 1.046]	6.5%
judge-selfpref	0.0	0.077 [-0.366, 0.497]	0.0	0.155 [-0.486, 0.782]	0.5	1.643 [0.979, 2.915]	7.0%
judge-mixed	0.2	0.837 [0.279, 2.446]	0.4	1.203 [0.302, 2.732]	0.0	-0.959 [-2.703, 0.170]	24.5%

Builder-authored reference. It is not learner evidence and is never credited to your portfolio.

Rubric3 criteria

Criterion	Meets	Exceeds
Measurement validity	Human-human ceiling and length-rule floor reported; agreement with intervals.	Self-preference and contamination risks analysed with a small fresh-label check.
Bias analysis	Order flip rate and length bias measured on the frozen sample.	Shows how order balancing changes conclusions about which model wins.
Usage policy	States where the judge may replace humans.	Defines re-validation triggers and the human fallback for disagreements.

References4

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
MT-Bench, the human-judgment data, and documented judge biases.
Large Language Models are not Fair Evaluators
Order sensitivity of LLM judges and balanced-position calibration.
Holistic Evaluation of Language Models
Multi-metric evaluation framing beyond a single score.
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
Why public benchmark data may be memorised by the judge.

At a glance