AI Glossary Term

What does “ALiBi” mean?

Definition

A positional bias method that penalizes attention scores based on token distance, helping models extrapolate to longer contexts.

Related terms

RoPE (Rotary Positional Embeddings)
A positional encoding method that rotates query and key vectors to encode relative token positions.
Positional Encoding
Information added to token embeddings so transformers can distinguish sequence order.
Sliding Window Attention
An attention pattern where each token attends only to a fixed-size window of nearby tokens to reduce compute.
Algorithmic Bias
Systematic unfairness in model outputs caused by skewed data, assumptions, or modeling choices.
Attention Mechanism
A model component that dynamically focuses on relevant parts of an input when producing an output.
Bias
A consistent pattern of error or unfairness in data or model behavior.

Learn more in our free guides

ALiBi Position Bias

See also

BPE (Byte Pair Encoding)Multi-Head AttentionSentencePieceFlash AttentionANN (Approximate Nearest Neighbor)QLoRAHNSWDPO (Direct Preference Optimization)
Browse the full AI GlossaryExplore all free AI guides