LLMs, GenAI, RAG, agents, evaluation · insight-0666
Prefer exact-match response caching, and scope any semantic cache very narrowly
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- aviation, telecom
Summary: situation, action, reason, boundary
Situation
To save cost, the team proposes caching answers and serving them for new questions whose embeddings are similar.
Action
Cache exact normalised requests (including user scope and config version), use provider prompt caching for shared prefixes, and allow semantic caching only for static, non-personal FAQs with a high threshold and audited hit samples.
Reason
Embedding similarity ignores small differences in dates, numbers and entities that change the correct answer, so a semantic hit is a guess presented as a cached fact.
Boundary
High-volume assistants with repeated questions.
Symptom or risk
"Change fee for IST-FRA on 12 May" returns the cached answer for "IST-FRA on 21 May" or for another fare class, because the vectors are close.
Exceptions
Pure FAQ deflection where answers do not depend on parameters.
Trade-offs
Exact caching yields fewer hits; measure hit rate and error rate together.
Pitfalls
Caching across users or tenants, which can leak one customer's answer to another.