LLMs, GenAI, RAG, agents, evaluation · insight-1646
Write to a response cache only when the generation ended naturally and passed validation
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- ML engineer, Applied AI
- Industries
- public-sector, aviation
Summary: situation, action, reason, boundary
Situation
A response cache stores whatever came back from the model, keyed on the normalised request.
Action
Gate cache writes on a natural stop reason, passing validators and no safety flag, and never cache tool-using answers that depend on live data. Count cache hits per outcome class and expire entries with a time-to-live.
Reason
A max_tokens stop means the output was cut off and incomplete, and a cache converts one transient failure into a persistent one that is also cheap and fast to serve.
Boundary
Exact-match response caches in front of generation.
Symptom or risk
A truncated, schema-invalid or filter-blocked answer produced once is served to every later user who sends the same request, long after the cause is gone.
Exceptions
Caching an explicit, deliberate fallback message under a short time-to-live to absorb a retry storm.
Trade-offs
Stricter write gates lower the hit rate and need the validators to run before caching, adding a little latency.
Pitfalls
Caching refusals produced by a transient classifier outage, or keying only on text while ignoring the configuration that shaped the output.
Lessons and concepts
Claims and their sources
A max_tokens stop reason means the model output was cut off and is incomplete.