Structured Pruning and Layer Dropping Quiz

Answers and explanations appear as you take the quiz.

1. Why does structured pruning produce real speedups while unstructured pruning often does not?
   • Structured pruning removes coherent blocks that actually shrink the tensors
   • Unstructured pruning increases the model size
   • Structured pruning adds more layers
   • Unstructured pruning only works on CPUs
2. What is 'layer dropping' in the context of transformers?
   • Reducing the learning rate per layer
   • Deleting entire transformer blocks to reduce model depth
   • Adding dropout to every layer
   • Reordering the layers randomly
3. Which signal commonly indicates that a transformer layer can be safely dropped?
   • The layer has the most parameters
   • The layer is the first in the network
   • The layer's input and output hidden states are very similar
   • The layer uses the most memory
4. What step typically recovers accuracy after structured pruning?
   • Increasing the batch size only
   • Re-quantizing to 16 bits
   • Adding more attention heads
   • A short round of fine-tuning or knowledge distillation
5. How are individual attention heads often ranked for pruning?
   • By the increase in loss when the head is masked (sensitivity)
   • By alphabetical order
   • By their position in the vocabulary
   • By the number of tokens they have seen
6. What is a major practical use of structured pruning combined with distillation?
   • Generating training data automatically
   • Deriving a family of smaller, faster models from one large checkpoint
   • Encrypting model weights
   • Increasing the context window length
7. Why can a smaller student recover some function of a pruned model during distillation?
   • It restores the deleted weights in place
   • It changes the test labels
   • It is trained to match useful outputs or behavior of the larger teacher
   • It removes the need for validation
8. If a transformer layer’s input and output have high cosine similarity, what does that suggest under the guide’s pruning heuristic?
   • The layer is guaranteed to be useless for every task
   • The model must add another layer
   • The output has no features
   • The layer changed the representation relatively little and may be a candidate for dropping