AI Glossary Term

What does “KV Cache” mean?

Definition

Stored key and value tensors from previous tokens that let transformers generate new tokens without recomputing past attention.

Browse the full AI Glossary Explore all free AI guides

How KV Cache works

Example

When a chatbot writes a long reply, it does not recompute attention inputs for every earlier token at each step. It keeps those stored values and only adds those for the newest token.

Why it matters

The cache makes generation much faster, but it uses memory that grows with the conversation length. This is a major factor in serving cost for long contexts.

Common questions

What are keys and values?

Vectors each token produces in attention layers that later tokens use to decide what to focus on.

Why does it limit long contexts?

Cache size grows with the number of tokens, so memory becomes a bottleneck.

How is the cost reduced?

Through quantizing the cache, sharing keys and values across attention heads, and limiting attention windows.

Continual LearningTraining approaches that let a model keep learning from new data without forgetting prior knowledge.Top-k SamplingA decoding strategy that samples only from the k most likely next tokens.Unsupervised LearningLearning patterns from unlabeled data without explicit target outputs.Sliding Window AttentionAn attention pattern where each token attends only to a fixed-size window of nearby tokens to reduce compute.SentencePieceA language-agnostic tokenizer that learns subword units directly from raw text without pre-splitting on whitespace.Presence PenaltyA decoding setting that reduces the probability of tokens that have appeared at all, encouraging new topics.

Learn more in our free guides

KV Cache KV Cache Optimization Cache-Augmented Generation (CAG)

Test yourself

KV Cache Quiz KV Cache Optimization Quiz Cache-Augmented Generation (CAG) Quiz