LLMs, GenAI, RAG, agents, evaluation · insight-0610
Stop adding retrieved passages once answer quality stops rising
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Applied AI, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Answers miss facts, and the proposed fix is to raise k from 20 to 50 because the context window allows it.
Action
Plot evidence recall and answer quality against k on the golden set, and choose the smallest k after which answer quality flattens.
Reason
Liu and colleagues found that reader performance saturated long before retriever recall: moving from 20 to 50 retrieved documents improved accuracy only by about 1 to 1.5 percent while greatly lengthening the input.
Boundary
Open-domain and enterprise QA where the retriever returns many partially relevant passages.
Symptom or risk
Latency and token cost rise steeply while answer quality barely moves, because the model does not use the extra passages.
Exceptions
Synthesis tasks that genuinely need many sources, such as summarising all incident reports for a route; budget for them explicitly.
Trade-offs
A smaller k risks missing rare evidence; improve ranking rather than padding the context.
Pitfalls
Reading recall@50 as proof that the model will use passage 47.
Lessons and concepts
Claims and their sources
In an open-domain QA case study, reader performance saturated long before retriever recall, and using 50 instead of 20 retrieved documents improved accuracy only marginally (about 1 to 1.5 percent, for the two models tested).