LLMs, GenAI, RAG, agents, evaluation · insight-1519
Re-check the minimum cacheable prompt length whenever you change models
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A 1,500-token system prompt cached well on one model, and the team moves the assistant to another model in the same family.
Action
Add a post-migration check that repeats a request and asserts non-zero cache read tokens, store the minimum length per model next to the prompt version, and lengthen the shared prefix only with content that truly belongs there.
Reason
Anthropic's documentation lists minimum cacheable lengths that differ by model, from 512 to 4,096 tokens when checked, and says a prompt below the minimum is not cached and raises no error.
Boundary
Any workflow that caches a shared prefix near the thresholds.
Symptom or risk
Caching silently stops, because prompts below a model's minimum are processed without error and without caching, and cost and latency jump after the migration.
Exceptions
Prefixes far above every threshold of the models you use.
Trade-offs
Per-model configuration and a smoke test per migration.
Pitfalls
Concluding that caching is not worth it after a test run that was below the minimum.
Lessons and concepts
Claims and their sources
Anthropic's prompt caching documentation states that minimum cacheable prompt length varies by model (examples read were 512, 1,024, 2,048 and 4,096 tokens), and that when both cache read and cache creation counts are zero the prompt was not cached, most likely because it was below the minimum, with no error.