Activation Recomputation Tradeoffs Quiz

Answers and explanations appear as you take the quiz.

1. What does activation recomputation trade away to save memory?
   • Model accuracy
   • Extra compute (FLOPs)
   • Network bandwidth
   • Numerical precision
2. Why are forward-pass activations normally stored at all?
   • To display them to the user
   • To compress the model
   • They are required for tokenization
   • Because backpropagation needs them to compute gradients
3. Why should activation checkpointing overhead be measured on the actual model?
   • Checkpoint placement, operators, and hardware determine the cost of replay
   • It always adds exactly forty percent to wall-clock time
   • It guarantees a tenfold speed improvement
   • It removes all forward computation
4. For an idealized N-layer chain with comparable activations, what storage scaling can square-root checkpoint placement achieve?
   • O(N squared) for all model parameters
   • O(1) for all training memory regardless of N
   • O(sqrt(N)) for activation storage
   • O(N squared) for optimizer state
5. What is the idea behind selective (not full) recomputation?
   • Recompute every operation twice
   • Never recompute anything
   • Recompute only the embedding layer
   • Recompute only memory-heavy but compute-cheap ops, cache expensive ones
6. Which complementary technique is often combined with recomputation to save even more memory?
   • Increasing the batch size
   • Offloading activations to CPU or NVMe
   • Removing all normalization layers
   • Switching to inference mode
7. A checkpointed block contains dropout. Why does consistent RNG state matter during replay?
   • Dropout becomes a model-weight quantizer
   • The optimizer no longer needs gradients
   • Backward recomputation should reproduce the intended forward randomness
   • The batch size must become one
8. Checkpointing reduces stored activations, but optimizer states still exceed GPU memory. Why is the job still out of memory?
   • Activation recomputation does not remove optimizer-state storage
   • Checkpointing increases every optimizer tensor to FP64
   • Backward no longer consumes any memory
   • The model cannot use checkpointing with an optimizer