KV Cache
The KV cache stores the key and value vectors a transformer has already computed for previous tokens, so it doesn't have to recompute them for every new word it generates.
Overview
It is the single biggest reason text generation is fast — and the main thing eating your GPU memory during long conversations.
Deep Dive
Transformers generate text one token at a time, and each new token's attention layer needs to compare against every prior token. The attention mechanism turns each token into a query, key, and value vector. Without caching, generating token number 1,000 would mean recomputing keys and values for all 999 earlier tokens at every step — quadratic, wasteful work. The KV cache saves those key and value vectors after they're first computed and reuses them, so each new step only computes vectors for the single newest token and attends over the stored cache. This shrinks per-token cost from scaling with sequence length to roughly constant. The trade-off is memory: the cache grows linearly with context length, number of layers, and attention heads, often becoming the dominant memory consumer in long-context serving.
Technical Insight
During the 'prefill' phase the model processes the whole prompt and fills the cache; during 'decode' it appends one token's K/V per step and reattends. Cache size scales as 2 (K and V) × layers × heads × head_dim × sequence_length × batch, in the chosen precision. To tame this, modern models use grouped-query or multi-query attention to share keys/values across heads, and serving systems like vLLM use PagedAttention to allocate cache in non-contiguous blocks, cutting fragmentation and waste.
Strategic Impact
Speed and scale
Language workflows can move faster without sacrificing consistency.
Access and reach
It expands access across languages and communication styles.
Clearer decisions
Teams can spend more time on judgment while automation handles repetition.
The Future of KV Cache
As context windows stretch into the hundreds of thousands of tokens, the KV cache becomes the central bottleneck, so innovation is fierce: cache quantization to 8 or 4 bits, eviction policies that drop low-importance tokens, cross-request prefix sharing, and offloading to CPU or disk. Architectural shifts like multi-head latent attention compress the cache itself. Expect continued co-design of attention variants and memory systems aimed at serving very long contexts cheaply and at high throughput.
Real-World Implementation
Speeding up chatbot replies by reusing cached keys/values from the conversation history instead of reprocessing it each turn.
Prefix caching that shares the cache for a long system prompt across many users, cutting cost and latency.
vLLM's PagedAttention managing KV cache in blocks to serve many concurrent requests on one GPU efficiently.
Quantizing the KV cache to lower precision to fit longer contexts into limited GPU memory.
Risks & Guardrails
Hallucinated facts can quietly enter reports, support flows, or research outputs.
Prompt sensitivity can create inconsistent results across similar requests.
Sensitive text data may be exposed if access controls are weak.
Implementation Roadmap
Define output format, tone, and quality standards before rollout.
Ground responses with trusted sources whenever accuracy matters.
Keep a human review checkpoint for high-stakes outputs.
Track failure patterns and retrain prompts or workflows regularly.
Keep Exploring
Free newsletter
Keep up with AI in 3 minutes a day
One short email each weekday with the three AI stories that actually matter. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the KV Cache quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Next guide
KV Cache Optimization
Frequently asked questions
What is KV Cache?
The KV cache stores the key and value vectors a transformer has already computed for previous tokens, so it doesn't have to recompute them for every new word it generates. It is the single biggest reason text generation is fast — and the main thing eating your GPU memory during long conversations.
What does the KV cache store?
The KV cache holds the key and value vectors from earlier tokens so attention can reuse them instead of recomputing.
What problem does the KV cache primarily solve?
Without caching, each new token would require recomputing K/V for every previous token, which is wasteful; the cache makes per-token cost roughly constant.
What is the main downside of the KV cache?
The cache grows with sequence length, layers, and heads, often becoming the dominant consumer of GPU memory in long-context serving.
Which technique reduces KV cache size by sharing keys and values across attention heads?
Grouped-query and multi-query attention let multiple query heads share fewer key/value sets, shrinking the cache substantially.
What are the two phases of transformer inference relative to the KV cache?
Prefill fills the cache with the prompt's K/V; decode appends one new token's K/V each step and reattends over the cache.