LLMs, GenAI, RAG, agents, evaluation · insight-0617
Use MRR when one right passage exists and how high it ranks matters
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Applied AI, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Each question has a single authoritative passage, such as one clause of a contract, and you want one number for ranking quality.
Action
Report mean reciprocal rank: for each question take 1/rank of the first relevant passage (0 if none in the cutoff), then average.
Reason
MRR, as used in the TREC-8 question answering track, rewards putting the first correct response at the top while staying bounded between 0 and 1.
Boundary
Lookup-style questions, FAQ matching, single-answer support search.
Symptom or risk
Recall@5 treats rank 1 and rank 5 the same, so a reranker that moves evidence from 4th to 1st shows no gain.
Exceptions
Questions needing several passages; MRR only sees the first hit, so use recall of all required evidence or nDCG.
Trade-offs
MRR is dominated by the top position; a drop from rank 1 to 2 halves the per-question score, which can exaggerate small changes.
Pitfalls
Averaging MRR over questions with no labelled evidence, which silently counts them as zero.
Lessons and concepts
Claims and their sources
In the TREC-8 QA track, each question scored the reciprocal of the rank of the first correct response among five (0 if none), and the run score was the mean.