KV cache
Also known as: KV cache · key-value cache
The KV cache stores the key and value vectors that attention computed for tokens already generated. When producing the next token, the model reuses them instead of recomputing the whole prefix.
It’s a huge speedup — but it costs memory, and it grows with the context length.
- Speed win: avoid recomputing past tokens.
- Memory cost: cache size ≈ tokens × layers × heads.
Why long context is expensive: the KV cache is a big chunk of the memory a long conversation consumes. It's one reason very long contexts are slow and memory-hungry.