LLMs, GenAI, RAG, agents, evaluation · insight-0616
Report recall@k at the k you actually pass to the model
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Applied AI, Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A retrieval dashboard shows recall@100, while the generation prompt receives the top 5 chunks.
Action
Compute recall@k (share of questions whose labelled evidence appears in the top k) at the production k, alongside a larger k that shows reranking headroom.
Reason
Ranked-retrieval measures are defined over a cutoff; the cutoff that matters for RAG is the context the model receives, not the candidate pool.
Boundary
Every RAG evaluation on days 51 and 57.
Symptom or risk
Retrieval looks excellent and answer failures are blamed on the LLM, although the evidence ranked 6th to 100th never reached it.
Exceptions
Multi-evidence questions need a stricter variant: all required passages in the top k, not just one.
Trade-offs
Recall@k ignores ordering inside the top k; pair it with MRR or nDCG when position matters.
Pitfalls
Changing k between experiments and comparing the recall numbers directly.
Example
Synthetic illustration: recall@100 = 0.94 but recall@5 = 0.61 means about 39% of questions fail before generation starts.
Lessons and concepts
Claims and their sources
Ranked-retrieval evaluation uses measures defined at a cutoff, such as precision at k, alongside rank-sensitive measures such as MAP and NDCG.