LLMs, GenAI, RAG, agents, evaluation · insight-1549
Run an oracle-context test to separate the retrieval gap from the generation ceiling
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
RAG answer accuracy is 62% and the team splits between improving retrieval and improving the prompt without agreement.
Action
Run the same questions three ways: no context, retrieved context, and the gold evidence passages supplied directly; compare the three scores on the same items and attribute the gaps.
Reason
Gold-context accuracy bounds what better retrieval can deliver, and the distance from there to 100% is generation or labelling error; the retrieved-versus-gold gap is the retrieval headroom.
Boundary
Any RAG evaluation with labelled evidence passages.
Symptom or risk
Work goes into the stage with the smaller headroom, because the end-to-end score alone does not show how much each stage can still gain.
Exceptions
Questions needing synthesis across many documents, where a single gold passage is not a meaningful oracle.
Trade-offs
Needs gold evidence labels and three evaluation passes.
Pitfalls
A high no-context score signals contamination or parametric knowledge; the retrieval gain may then be small for reasons unrelated to retrieval.
Example
Hypothetical on 200 questions: no context 35%, retrieved 62%, gold 86%. Retrieval headroom is 24 points; the generation ceiling leaves 14 points.