ALiBi Position Bias Quiz

Answers and explanations appear as you take the quiz.

1. What does ALiBi stand for?
   • Attention with Linear Biases
   • Adaptive Linear Bias
   • Aligned Bidirectional Encoding
   • Absolute Linear Indexing
2. How does ALiBi penalize distant tokens?
   • By zeroing out embeddings beyond a fixed window
   • By adding a penalty proportional to the distance between query and key positions
   • By learning a separate embedding for every position
   • By rotating the query and key vectors
3. What is ALiBi's most celebrated practical advantage?
   • It eliminates the need for a softmax
   • It makes attention run in constant memory
   • It extrapolates well to sequences longer than those seen in training
   • It removes the need for multiple attention heads
4. What is different about the linear bias across attention heads?
   • Each head shares the exact same slope
   • The slopes are learned during training
   • Only the first head receives a bias
   • Each head gets its own fixed slope, forming a geometric sequence
5. Compared to traditional positional embeddings, ALiBi adds how many learned parameters?
   • None — the bias slopes are fixed, not learned
   • One per position
   • One per token
   • Two per attention head
6. Which well-known large model used ALiBi for position information?
   • GPT-2
   • BLOOM
   • BERT
   • Word2Vec
7. At what stage does ALiBi apply its distance-based bias?
   • To token IDs before they enter the embedding table
   • To the final vocabulary probabilities after decoding
   • To query-key attention scores before the softmax
   • To the training labels after loss calculation
8. Why can ALiBi support inference on sequences longer than those used in training?
   • It removes the need for attention at longer lengths
   • It automatically retrains the model while generating
   • It stores a complete answer for every possible sequence length
   • Its distance-based bias does not require a separately learned embedding for every absolute position