Attention Rollout and Head Pruning Quiz

Answers and explanations appear as you take the quiz.

1. What is the goal of attention rollout?
   • To trace cumulative information flow from input tokens to outputs across layers
   • To add more layers to a Transformer
   • To increase the learning rate
   • To convert attention into convolution
2. Why is a single layer's attention map often insufficient for explanation?
   • Attention is never computed
   • Information mixes across many layers and heads, so one map misses the full path
   • Maps are always uniform
   • Transformers have only one head
3. What does attention rollout add to each attention matrix to account for residual connections?
   • A bias of zero
   • A random noise term
   • An identity matrix component before normalizing
   • An extra softmax
4. What did research on multi-head attention reveal about head redundancy?
   • Every head is essential
   • Heads cannot be removed
   • Pruning always doubles accuracy
   • Many heads can be pruned with little accuracy loss
5. How is a head's importance commonly estimated for pruning?
   • Via gradient-based sensitivity of the loss to a head mask
   • By its alphabetical position
   • By counting its parameters only
   • By the time of day
6. What different jobs do attention rollout and head pruning each do?
   • They are identical techniques
   • Rollout aids interpretation while pruning exploits redundancy to compress models
   • Both only add parameters
   • Neither touches attention
7. In an attention-rollout calculation, why are residual paths included with layer attention?
   • To make every head have the same weight
   • To remove the value vectors from the Transformer
   • To approximate information that can pass along the identity path when layer transitions are composed
   • To prevent the attention matrix from being normalized
8. How should an attention-rollout score be interpreted?
   • As proof that a token caused the output
   • As the exact value vector used at every layer
   • As a measure of how many heads the model contains
   • As a post-hoc approximation of relative token influence, not a causal explanation