LLMs, GenAI, RAG, agents, evaluation · insight-1539
Choose rerank depth from the recall-at-N curve against reranker latency, not from a round number
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
The reranker scores the top 100 candidates because that was the tutorial default, and latency is over budget.
Action
Measure candidate recall at N = 10, 20, 50, 100 on labelled queries, measure reranker time per N at your passage length, and pick the smallest N where recall is within a chosen margin of the plateau.
Reason
A cross-encoder scores each query-passage pair separately, so cost grows with candidates and passage length, while the first-stage recall curve flattens as N grows.
Boundary
Two-stage retrieval with a cross-encoder or LLM reranker.
Symptom or risk
The system pays for candidates deep in the list that almost never contain the missing right passage, or cuts depth blindly and loses recall the reranker could have used.
Exceptions
Rerankers that are cheap per pair on idle GPUs with no latency target.
Trade-offs
A smaller N is cheaper but caps what reranking can fix; recall must be rechecked after a retriever change.
Pitfalls
Truncating passages to speed up scoring, which can drop the sentence that holds the answer.
Example
Synthetic curve: recall@20 0.78, @50 0.88, @100 0.91, with rerank time of 80, 200 and 400 ms. Going from 50 to 100 buys 3 points for 200 ms.
Lessons and concepts
Claims and their sources
The Sentence Transformers retrieve-and-rerank documentation says a cross-encoder scores each query-passage pair, which is too slow for a whole collection, so it is applied to a small candidate set from the retriever (the page's example is about 100).