LLMs, GenAI, RAG, agents, evaluation · insight-1627
Test the whole corpus in a cached prompt before building retrieval for a small stable knowledge base
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Applied AI, ML engineer
- Industries
- aviation, insurance
Summary: situation, action, reason, boundary
Situation
A product assistant must answer from a policy handbook of a few hundred pages that changes a few times a year, and the plan starts with a vector index.
Action
Count the corpus tokens. If it fits the window, run the golden set with the whole corpus as a cached prefix and compare accuracy, cost and latency with a RAG baseline; keep retrieval only if it wins.
Reason
Anthropic's 2024 retrieval write-up (a vendor source) says a knowledge base under about 200,000 tokens can be placed directly in the prompt, and that prompt caching makes this faster and cheaper.
Boundary
Small, slowly changing corpora served at moderate to high query volume.
Symptom or risk
The team maintains chunking, embeddings and reranking, with failure modes a simpler design would not have.
Exceptions
Corpora that exceed the window, need per-user document permissions, or change faster than the cache lifetime.
Trade-offs
Every query still pays the cache-read price of the whole corpus, and long contexts need their own quality check at your input lengths.
Pitfalls
Trusting the vendor figures without a golden-set comparison, or forgetting that expired cache entries are rewritten at full price.
Lessons and concepts
Claims and their sources
Anthropic's Contextual Retrieval post (a vendor source) says a knowledge base smaller than about 200,000 tokens can be included in the prompt instead of using RAG, and that prompt caching makes this cheaper and faster.
A prompt cache hit requires an identical prefix up to the breakpoint, and cache entries have a limited lifetime that is refreshed on each hit.