What does Sliding Window Attention mean? Definition | AI Understanding

Definition

An attention pattern where each token attends only to a fixed-size window of nearby tokens to reduce compute.

Related terms

Removing less important model weights or neurons to reduce size and compute.

Usage-Based Billing

Pricing where costs scale with API calls, tokens, inference time, or consumed compute.

An empirical relationship showing how performance improves with model size, data, or compute.

A model where many parameters are zero or inactive to reduce computation.

Speculative Decoding

An inference acceleration method where a small draft model proposes tokens that a larger model verifies in parallel.

The maximum amount of input tokens a language model can process at once.

Learn more in our free guides

Attention Rollout and Head Pruning Sequence Parallelism and Ring Attention Multi-Head Latent Attention RWKV Linear Attention

See also

ALiBi BPE (Byte Pair Encoding)RoPE (Rotary Positional Embeddings)SentencePiece Positional Encoding ANN (Approximate Nearest Neighbor)Multi-Head Attention HNSW

Browse the full AI Glossary Explore all free AI guides