LLMs, GenAI, RAG, agents, evaluation · insight-0506
Estimate key-value cache memory before promising long contexts on self-hosted models
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
An ML engineer must serve a self-hosted model with long contexts to many concurrent users on a fixed GPU budget.
Action
Compute cache bytes per token as 2 x layers x key-value heads x head dimension x bytes per value, multiply by tokens and concurrent sequences, and add it to weight memory before committing to context length and concurrency.
Reason
Each active request keeps keys and values for all previous tokens, and this cache grows with length and concurrency, limiting batch size and throughput.
Boundary
Self-hosted inference sizing; comparing models with full multi-head versus grouped-query attention.
Symptom or risk
The weights fit, but under peak load the server runs out of memory or cuts batch size and throughput collapses.
Exceptions
Hosted APIs hide this; you see it as latency, rate limits and price.
Trade-offs
Grouped-query attention and cache quantization save memory but must be checked for quality loss on your eval set.
Pitfalls
Sizing hardware from parameter count alone.
Example
A hypothetical model with 32 layers, 8 key-value heads, head dimension 128 and 16-bit values needs 2 x 32 x 8 x 128 x 2 = 131,072 bytes per token, about 1 GiB for one 8,000-token sequence.
Lessons and concepts
Claims and their sources
Kwon et al. describe each request's key-value cache as large and dynamically growing, and show that wasted cache memory limits batch size and throughput.
Ainslie et al. show that sharing key-value heads across query heads (grouped-query attention) speeds decoding with quality close to full multi-head attention.