Multi-Query Attention Quiz

Answers and explanations appear as you take the quiz.

1. What does Multi-Query Attention share across all attention heads?
   • A single set of keys and values
   • The query projections
   • The softmax temperature
   • The output layer weights
2. What is the primary bottleneck that MQA addresses during autoregressive generation?
   • Floating-point compute throughput
   • Memory bandwidth from loading the KV cache
   • The number of training epochs
   • Tokenizer vocabulary size
3. By roughly what factor does MQA shrink the KV cache?
   • By the sequence length
   • By the batch size
   • By the number of attention heads
   • It does not change the KV cache size
4. What is the main trade-off of using MQA?
   • Much slower inference
   • It can no longer use multiple heads at all
   • It requires far more training data
   • A modest drop in model quality
5. Which variant sits between standard multi-head attention and MQA?
   • Grouped-Query Attention
   • Sliding-Window Attention
   • Flash Attention
   • Sparse Attention
6. Who proposed Multi-Query Attention?
   • Geoffrey Hinton
   • Noam Shazeer
   • Yann LeCun
   • Andrej Karpathy
7. In Multi-Query Attention, what remains separate for each attention head?
   • The shared key projection
   • The shared value projection
   • The query projection
   • The token sequence itself
8. How does Grouped-Query Attention differ from standard MQA?
   • It shares query projections but gives every head separate keys and values
   • It removes attention and replaces it with a recurrent layer
   • It groups output vocabulary tokens instead of attention heads
   • It shares keys and values within several groups of query heads rather than across all heads at once