Attention Mechanisms Quiz

Answers and explanations appear as you take the quiz.

1. What does token attention combine when forming an output representation?
   • Permanent copies of the entire conversation on disk
   • Value information from allowed positions using computed weights
   • Only manually chosen keywords with no learned scores
   • A translation dictionary that removes every other token
2. Which landmark 2017 paper introduced the transformer architecture?
   • 'Deep Residual Learning for Image Recognition'
   • 'Playing Atari with Deep Reinforcement Learning'
   • 'Attention Is All You Need'
   • 'Generative Adversarial Networks'
3. What are the three vectors computed for each token in attention?
   • Query, key, and value
   • Input, output, and bias
   • Mean, median, and mode
   • Encoder, decoder, and loss
4. In the formula softmax(QK^T / √d_k) V, why divide by √d_k?
   • To convert the result into integers
   • To remove negative numbers
   • To make the model run on a GPU
   • To keep the dot-product scores from becoming too large before softmax
5. Why does dense self-attention involve growing pairwise work for very long token sequences?
   • Because it must download new data from the internet
   • Because longer text always contains rarer words
   • Because cost grows with the square of the sequence length (O(n²))
   • Because it switches to a slower programming language
6. What capacity does multi-head attention provide through separate learned projections?
   • A guarantee that every head has one unique named meaning
   • Several attention representations that can capture different relationships in parallel
   • Removal of every token embedding
   • A reduction of vocabulary size by definition
7. In a scalar attention example, values two and six receive weights three-quarters and one-quarter. What is their weighted output?
   • Eight
   • Four
   • Two
   • Three
8. What does a causal attention mask prevent in an autoregressive text decoder?
   • Using later token positions when forming the current position's representation
   • Using any previous token
   • Computing any query or key vectors
   • Learning all model parameters