LLMs, GenAI, RAG, agents, evaluation · insight-0609
Retrieve wide with a cheap retriever, then rerank narrow with a cross-encoder
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Applied AI, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
The right passage is often in the top 50 but rarely in the top 5 you pass to the model.
Action
Retrieve 50 to 100 candidates, rescore them with a cross-encoder reranker, and pass only the top few; tune the candidate count against latency.
Reason
A cross-encoder reads query and passage together and ranks more precisely than a bi-encoder, but is too slow to run over the whole corpus. Nogueira and Cho showed large ranking gains from BERT re-ranking of first-stage candidates.
Boundary
When recall at a large k is good but precision at small k is poor.
Symptom or risk
Recall@50 looks healthy while answers fail, because the evidence never reaches the context window.
Exceptions
If recall@100 itself is low, a reranker cannot help; fix chunking, filters or the first-stage retriever.
Trade-offs
Reranking cost grows with the number of candidates; BEIR notes the high computational cost of re-ranking models.
Pitfalls
Measuring the reranker only end to end, without recall@k before and after it.
Lessons and concepts
Claims and their sources
Re-ranking first-stage candidates with a BERT cross-encoder substantially improved passage ranking quality.
In BEIR, re-ranking models achieved the best average zero-shot performance but at high computational cost.