LLMs, GenAI, RAG, agents, evaluation · insight-1517
Hold per-request toggles constant inside a cached prefix, because they invalidate it
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A support agent sets tool_choice per turn, attaches an image on some turns and varies its thinking settings, and the cache hit rate is far below what prompt ordering predicts.
Action
Treat tool_choice, image presence, thinking parameters and similar toggles as part of the prefix; fix them per conversation or route, log cache read and creation tokens per toggle combination, and fork routes instead of flipping settings mid-conversation.
Reason
Anthropic's caching page lists tool definitions, tool_choice, images and thinking parameters among changes that invalidate cached levels; a change invalidates its level and every level after it.
Boundary
Tool-using or multimodal conversations with long shared prefixes.
Symptom or risk
Prefixes that look identical are rewritten at cache-write prices after every change, so cost and latency stay high.
Exceptions
Prompts with no cache breakpoints, and low-traffic routes where entries expire between calls.
Trade-offs
Route forking multiplies prompt variants and cache entries.
Pitfalls
Verifying cache hits only on a test that never flips a toggle.
Lessons and concepts
Claims and their sources
Anthropic's prompt caching documentation says a change at one level of the tools, system, messages hierarchy invalidates that level and all later ones, and lists tool definitions, tool_choice, images and thinking parameters among changes that invalidate cached content (as read on 2026-10-12).