LLMs, GenAI, RAG, agents, evaluation · insight-0625
Slice retrieval metrics by query type before declaring a retriever better
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Applied AI, Data analyst
- Industries
- telecom, cross-industry
Summary: situation, action, reason, boundary
Situation
A hybrid retriever raises average recall@5 by two points over the dense baseline.
Action
Tag golden questions by type (exact lookup, conceptual, comparison, multi-hop, follow-up, unanswerable) and report each metric per slice, with slice sizes.
Reason
Different retrievers fail on different query shapes; slice results tell you which component to change and which users a release affects.
Boundary
Any retrieval comparison with a mixed query population.
Symptom or risk
The average hides that identifier lookups improved sharply while multi-hop or conversational follow-up questions got worse.
Exceptions
Tiny golden sets where slices of five questions are too noisy; merge slices or label more.
Trade-offs
Slice tags must be assigned consistently; write the tagging rules down with the golden set.
Pitfalls
Choosing slices after seeing results, which turns slicing into a search for a favourable story.