LLMs, GenAI, RAG, agents, evaluation · insight-0607
Fuse lexical and vector results by rank, not by adding raw scores
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Applied AI, ML engineer
- Industries
- e-commerce, cross-industry
Summary: situation, action, reason, boundary
Situation
You combine BM25 and embedding search into a hybrid retriever and need one ranked list.
Action
Start with reciprocal rank fusion, which scores each document by summed 1/(c + rank) across lists, and only move to tuned score normalisation if a labelled set shows a gain.
Reason
Rank fusion ignores score scales entirely. Cormack and colleagues found that reciprocal rank fusion outperformed Condorcet fusion and individual learned rankers in their experiments.
Boundary
Any hybrid retrieval where component scores are not calibrated to each other.
Symptom or risk
Adding BM25 scores (unbounded, query-dependent) to cosine similarities (roughly bounded) lets one retriever dominate unpredictably from query to query.
Exceptions
When one retriever is clearly better on every slice, fusion can dilute it; check per-slice results.
Trade-offs
Fusion discards score magnitude, so a very confident lexical hit counts the same as a marginal one at the same rank.
Pitfalls
Tuning fusion weights on the same queries used to report the final result.
Lessons and concepts
Claims and their sources
Reciprocal rank fusion was reported to outperform Condorcet fusion and individual rank-learning methods.