AI Glossary Term

What is ALiBi?

Definition

A positional bias method that penalizes attention scores based on token distance, helping models extrapolate to longer contexts.

Related terms

RoPE (Rotary Positional Embeddings)
A positional encoding method that rotates query and key vectors to encode relative token positions.
Positional Encoding
Information added to token embeddings so transformers can distinguish sequence order.
Sliding Window Attention
An attention pattern where each token attends only to a fixed-size window of nearby tokens to reduce compute.
Algorithmic Bias
Systematic unfairness in model outputs caused by skewed data, assumptions, or modeling choices.
Attention Mechanism
A model component that dynamically focuses on relevant parts of an input when producing an output.
Bias
A consistent pattern of error or unfairness in data or model behavior.

Learn more in our free guides

ALiBi Position Bias
Browse the full AI GlossaryExplore all free AI guides