Introduced on day 51
Mean reciprocal rank (MRR)
The average over queries of 1 divided by the rank of the first relevant passage, counted as 0 when none appears within the cutoff.
In detail
MRR suits lookup-style questions with one authoritative passage and a need to rank it high. It sees only the first relevant hit, so it cannot reward finding a second required passage, and it is dominated by the top position: moving a hit from rank 1 to rank 2 halves that query's score. Queries without any labelled evidence must be excluded, not averaged in as zeros.