LLMs, GenAI, RAG, agents, evaluation · insight-0606
Keep a BM25 baseline when queries contain codes, identifiers or rare terms
- Evidence type
- empirical finding
- Difficulty
- easy
- Roles
- Applied AI, ML engineer, Data scientist
- Industries
- aviation, manufacturing
Summary: situation, action, reason, boundary
Situation
Users search with fare basis codes, error codes, part numbers or policy clause ids, and the team plans a dense-only retriever.
Action
Run BM25 as a baseline on the same golden set, slice results by query type, and keep a lexical component wherever identifier queries matter.
Reason
BM25 rewards exact term overlap weighted by rarity. In BEIR's zero-shot comparison, BM25 was a robust baseline and dense retrievers often underperformed it outside their training domain.
Boundary
Technical support, aviation operations, parts catalogues, legal references.
Symptom or risk
Dense embeddings can map near-identical codes such as "Y26" and "Y62" to very similar vectors, so exact-match lookups fail while semantic queries look fine.
Exceptions
Paraphrase-heavy questions with little shared vocabulary, where dense retrieval usually wins.
Trade-offs
Two indexes to maintain and a fusion step to tune.
Pitfalls
Comparing on an average that hides the identifier slice entirely.
Lessons and concepts
Claims and their sources
In BEIR's zero-shot evaluation across heterogeneous datasets, BM25 was a robust baseline and dense retrieval models often underperformed out of domain.
BM25 scores documents by term-frequency saturation, inverse document frequency and document-length normalisation.