LLMs, GenAI, RAG, agents, evaluation · insight-1705
Send one request to write the prompt cache before fanning out parallel requests that share a prefix
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A batch job sends 50 requests at once that all begin with the same 20,000-token policy document, with caching enabled.
Action
Send one request first and wait until its response has begun, then fan out the rest. Check cache-read and cache-write token counts in the usage fields per request, and keep the fan-out inside the cache lifetime.
Reason
Anthropic's documentation (as read on 2026-10-12) says a cache entry becomes available only after the first response begins, and that parallel requests needing hits should wait for it.
Boundary
Evaluation sweeps, batch enrichment and agents that spawn parallel workers over one large prefix.
Symptom or risk
Requests that start before the first one begins responding find no cache entry, so many of them write it at full or write-premium price and the expected savings do not appear.
Exceptions
Providers with different caching rules, and prefixes below the minimum cacheable length.
Trade-offs
One request's time to first token is added in front of the fan-out.
Pitfalls
Changing anything inside the prefix between the warm-up and the fan-out, or letting the lifetime lapse during a long first response.
Example
Hypothetical prices: base input 0.02 at base. Fifty simultaneous writes cost 50 x 1.25 x 1.25; one write then 49 reads costs 0.098 = $0.123, about ten times less.
Lessons and concepts
Claims and their sources
Anthropic's prompt caching documentation (as read on 2026-10-12) says a cache entry becomes available only after the first response begins, that the default lifetime is 5 minutes and refreshed on use, and that hits need an exact prefix match.