LLMs, GenAI, RAG, agents, evaluation · insight-1546
Validate an LLM relevance labeller against chance-corrected agreement with human labels before it expands the golden set
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Applied AI, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Labelling passages for retrieval evaluation is slow, and the team wants an LLM to grade query-passage pairs at scale.
Action
Have humans label a stratified sample with a written guideline, compute agreement and chance-corrected agreement (Cohen's kappa), inspect disagreements by query type, freeze the prompt and keep a human-labelled test slice.
Reason
Studies find LLM relevance labels can match third-party human labellers in one search setting but prompt wording changes accuracy, and perspectives work leaves reliability for evaluation open.
Boundary
Scaling relevance labels for recall@k, MRR and nDCG on a domain corpus.
Symptom or risk
The model's relevance notion differs from the product's, and raw percent agreement looks high because most pairs are relevant, so bad labels enter the golden set unnoticed.
Exceptions
Final high-stakes evaluation sets, which should stay human-labelled.
Trade-offs
Human sampling cost remains; it buys the right to use the cheaper labeller.
Pitfalls
Re-prompting until agreement rises on the same sample, which overfits the judge to the validation slice.
Example
Hypothetical: 88% raw agreement with both raters calling 85% of pairs relevant gives chance agreement 0.85x0.85+0.15x0.15 = 0.745, so kappa = (0.88-0.745)/(1-0.745) = 0.53, only moderate.
Lessons and concepts
Claims and their sources
Thomas et al. report that, with a carefully designed prompt, LLM relevance labels were as accurate as third-party human labellers against real-searcher gold labels, and that prompt wording and paraphrases affect accuracy.
Faggioli et al. present arguments for and against automatic LLM relevance judgments and frame whether such judgments can be used reliably to evaluate retrieval systems as an open question.