LLMs, GenAI, RAG, agents, evaluation · insight-0612

Drop weakly relevant chunks instead of always filling k slots

Situation
Retrieval always returns exactly k chunks, even for questions where only one or none is relevant.
Action
Apply a relevance cut-off (a reranker score threshold or a relative drop from the top score), allow fewer than k chunks, and allow zero, which routes to an abstention response.
Reason
Shi and colleagues showed that adding irrelevant information to a problem sharply reduced LLM accuracy, so irrelevant context is not neutral filler.
Boundary
Narrow factual QA over heterogeneous corpora.

Symptom or risk

Off-topic passages pull the answer toward the wrong product or route, or the model blends unrelated facts.

Exploratory or summarisation tasks where breadth is the point.

Thresholds tuned too high cause false abstentions; tune on a golden set that includes unanswerable questions.

Using raw cosine similarity as the threshold; its scale shifts with the embedding model and query length.

LLM accuracy on arithmetic word problems dropped substantially when irrelevant context was added to the problem description.
Large Language Models Can Be Easily Distracted by Irrelevant Context