AI Glossary Term
What does “ALiBi” mean?
Definition
A positional bias method that penalizes attention scores based on token distance, helping models extrapolate to longer contexts.
How ALiBi works
Example
In ALiBi, a token's attention to a word far behind it is reduced by a penalty that grows with distance. Nearby words matter most, and the pattern holds for lengths beyond those seen during training.
Why it matters
It helps models handle longer texts than they were trained on, without adding learned position vectors.
Common questions
What does ALiBi stand for?
Attention with Linear Biases, introduced in a 2021 paper by Press and colleagues.
Does it fully solve long context?
No. It helps extrapolation, but quality can still fall on very long inputs.
Does it add parameters?
No. The penalties follow a fixed pattern rather than being learned.