Introduced on day 42
KV cache
Stored attention keys and values of earlier tokens that let a model generate each new token without recomputing the whole context.
In detail
Its size grows linearly with context length, number of layers and key-value heads, and with the number of concurrent requests, so it often limits how many long conversations a GPU can serve. Techniques such as grouped-query attention and paged memory management reduce or reorganise it.