LLMs, GenAI, RAG, agents, evaluation · insight-1545
Count unjudged retrieved passages as unknown, and judge a new retriever's top-k before comparing it
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Applied AI, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
The golden set's relevant passages were labelled from the old retriever's top 10, and a new retriever is scored against it.
Action
Collect the union of both systems' top-k, send passages with no label for judgment, add the verdicts to the set, and report the share of unjudged results at k next to recall.
Reason
Test collections judge only documents that some system returned, as TREC assessors did, so metrics computed with missing labels are biased toward the contributing systems.
Boundary
Comparing retrievers, chunkers or embedding models on a reused labelled set.
Symptom or risk
A truly relevant passage the old system never surfaced has no label, so the new retriever is penalised for finding it and the comparison favours the system that built the labels.
Exceptions
Exhaustively labelled corpora, which are small enough to judge every passage.
Trade-offs
More judgments cost reviewer or LLM-judge time; sample if the unjudged tail is large.
Pitfalls
Reporting a lower recall as a regression without checking that the "misses" are simply unlabelled.
Example
Synthetic: 3 labelled relevant passages for a query; the new system returns 4 relevant ones, one unlabelled. Naive recall@10 is 3/3, so the extra find is invisible, and among those four results precision looks like 3/4 instead of 4/4 purely because of the missing label.
Lessons and concepts
Claims and their sources
The Introduction to Information Retrieval book notes that test-collection relevance judgments exist only for documents that were among the top k returned by some system, rather than being exhaustive.