AI Glossary Term

What does “Sliding Window Attention” mean?

Definition

An attention pattern where each token attends only to a fixed-size window of nearby tokens to reduce compute.

Browse the full AI Glossary Explore all free AI guides

How Sliding Window Attention works

Example

With a window of 4,096, a token at position 50,000 attends only to the previous 4,096 tokens. Stacked layers still pass information further, so earlier content can influence it indirectly.

Why it matters

It makes long inputs cheaper to process. Details outside the window can be lost unless layers carry them forward.

Common questions

Which models use it?

Several open models, including some from Mistral and Longformer-style designs, use windowed attention.

Does the model forget old text?

Direct access is limited, though information can propagate through layers.

Window size vs context length?

The window limits direct attention; total context can be larger because layers stack.

PruningRemoving less important model weights or neurons to reduce size and compute.Usage-Based BillingPricing where costs scale with API calls, tokens, inference time, or consumed compute.Scaling LawAn empirical relationship showing how performance improves with model size, data, or compute.Sparse ModelA model where many parameters are zero or inactive to reduce computation.Speculative DecodingAn inference acceleration method where a small draft model proposes tokens that a larger model verifies in parallel.Context WindowThe maximum amount of input tokens a language model can process at once.

Learn more in our free guides

Attention Rollout and Head Pruning Sequence Parallelism and Ring Attention Multi-Head Latent Attention RWKV Linear Attention

Test yourself

Attention Rollout and Head Pruning Quiz Sequence Parallelism and Ring Attention Quiz Multi-Head Latent Attention Quiz