LLMs, GenAI, RAG, agents, evaluation · insight-1506
Have the model quote the relevant evidence before it answers from a long input
- Evidence type
- empirical finding
- Difficulty
- hard
- Roles
- Applied AI, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Retrieval is perfect in a 40,000-token prompt, yet multi-step answers over the supplied evidence are worse than on a trimmed version of the same prompt.
Action
Add a first step that asks the model to extract or recite the passages it needs, then answer from that short extract in the same or a second call, and compare with single-pass answering at your real lengths.
Reason
Du et al. report performance drops of 13.9% to 85% as input length grew despite perfect retrieval, even with irrelevant tokens masked, and that reciting the evidence first helped (up to 4% on RULER for GPT-4o).
Boundary
Reasoning, maths or question answering over long supplied context.
Symptom or risk
The team keeps tuning retrieval and chunking while the loss comes from the length of the input itself.
Exceptions
Short inputs and lookups answered by a single passage.
Trade-offs
Extra output tokens and latency, and recited text can itself contain errors that propagate.
Pitfalls
Treating the recited extract as verified evidence without checking it against the source.
Lessons and concepts
Claims and their sources
Du et al. report that with perfect retrieval, performance still dropped by 13.9% to 85% as input length grew within claimed limits across five models and three task types, and that reciting retrieved evidence before solving improved GPT-4o by up to 4% on RULER.