LLMs, GenAI, RAG, agents, evaluation · insight-1527
Gate an embedding or chunking change on logged-query replay, and read the queries whose top results changed most
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A candidate index (new embedding model, new chunker or new analyzer) beats the current one on the golden set and the team wants to switch.
Action
Replay a sample of recent production queries against both indexes, rank the queries by how much their top-k changed, read the most-changed ones with the old and new passages side by side, and only then switch.
Reason
Aggregate metrics hide where the change hurts; a diff of retrieved lists is cheap, needs no labels, and points a reviewer at the regressions and at the unlabelled queries worth adding to the golden set.
Boundary
Any retrieval change that is hard to roll back or touches every query: re-embedding, re-chunking, new fusion settings.
Symptom or risk
A golden set of a few hundred queries misses entire query families, so the aggregate recall@k rises while a frequent query type quietly loses its best passage.
Exceptions
Brand-new systems with no query logs; there, use synthetic or hand-written queries and say so.
Trade-offs
Reviewing diffs costs analyst time, and logged queries may need redaction before people read them.
Pitfalls
Sorting by overlap alone: a changed ranking is not a worse ranking, so each reviewed pair needs a verdict that feeds back into labels.
Lessons and concepts
Claims and their sources
Qdrant's migration guide warns that a new embedding model is not automatically better on your data and advises measuring it before moving traffic.