Gradient Accumulation Quiz

Answers and explanations appear as you take the quiz.

1. What does gradient accumulation primarily let you do?
   • Train without any GPU
   • Skip the backward pass
   • Simulate a larger effective batch size on limited memory
   • Eliminate the optimizer entirely
2. With micro-batch size 4 and 8 accumulation steps, the effective batch size is:
   • 32
   • 4
   • 8
   • 12
3. With an accumulation factor N, when should optimizer.step() and zero_grad() occur?
   • After every micro-batch
   • Never
   • Before the forward pass
   • Call optimizer.step() after N micro-batches, then clear gradients before the next accumulation window
4. To keep the effective learning rate correct, the per-micro-batch loss is usually:
   • Multiplied by N
   • Divided by the number of accumulation steps N
   • Set to zero
   • Replaced with accuracy
5. Which normalization layer pairs more cleanly with gradient accumulation?
   • Layer or group normalization, since they don't depend on batch statistics
   • Batch normalization, because it loves tiny batches
   • No normalization works with it
   • Only dropout
6. The main cost of using gradient accumulation is:
   • More GPU memory used at peak
   • Loss of all gradients
   • Longer wall-clock time per effective update
   • Inability to use an optimizer
7. A training loop clears gradients after every micro-batch but steps the optimizer every fourth batch. What is lost?
   • The model weights are deleted
   • The earlier micro-batch contributions to the accumulated update
   • The effective batch becomes four times larger
   • The loss automatically becomes zero
8. Why may four micro-batches with BatchNorm behave differently from one physical batch of the same total size?
   • The optimizer cannot accept four backward passes
   • Gradient addition is forbidden in BatchNorm
   • The number of training examples changes
   • BatchNorm uses statistics from each micro-batch rather than the combined batch