LLMs, GenAI, RAG, agents, evaluation · insight-0608

Choose the embedding model on your own labelled queries, not a public leaderboard

Situation
The team picks an embedding model because it tops a public retrieval benchmark.
Action
Shortlist two or three models plus BM25, embed the same corpus, and compare recall@k and nDCG on 100 to 300 labelled in-domain queries before committing.
Reason
BEIR showed that retriever rankings shift across datasets and domains, so a public benchmark average is weak evidence for your own corpus.
Boundary
Any new RAG corpus, especially non-English, technical or regulated text.

Symptom or risk

The leaderboard winner underperforms on your domain jargon, languages or document style, and nobody notices until users complain.

Prototypes where the retriever is not yet the bottleneck; even then, record the choice as provisional.

Re-embedding a large corpus for each candidate costs compute; use a representative sample first.

Switching embedding models later without re-indexing everything; vectors from different models are not comparable.

BEIR evaluated retrievers zero-shot on heterogeneous datasets and found that in-domain strength did not reliably transfer, with dense models often trailing BM25 out of domain.
BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models