LLMs, GenAI, RAG, agents, evaluation · insight-0670
Order prompts stable-first so provider prompt caching can actually hit
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Requests share a long system prompt, tool definitions or a reference document, but the cache-hit rate is near zero.
Action
Put the stable parts first (tools, system instructions, static documents) and the variable parts last; keep tool definitions identical across calls; check the cache read and write token counts in responses.
Reason
Anthropic's prompt caching needs an identical prefix in the order tools, system, messages; changing an earlier level invalidates everything after it. Cache hits cut input cost and time-to-first-token.
Boundary
RAG with shared instructions, agents with fixed toolsets, repeated questions over one document.
Symptom or risk
A timestamp, user name or request id near the top of the prompt changes the prefix on every call, so cached reads never happen and long-prompt latency stays high.
Exceptions
Low-traffic endpoints whose calls are further apart than the cache lifetime.
Trade-offs
Cache writes can cost more than ordinary input; caching pays only when reuse is frequent.
Pitfalls
Reordering retrieved chunks or tools nondeterministically between calls.
Lessons and concepts
Claims and their sources
Anthropic's prompt caching requires an identical prompt prefix, caches in the order tools, system, messages, and a change at one level invalidates that level and those after it.
Cache hits reduce processing time and time-to-first-token, and cache reads are billed at a fraction of the base input price.