ZeRO and Sharded Optimizers Quiz

Answers and explanations appear as you take the quiz.

1. What redundancy does ZeRO eliminate compared to plain data parallelism?
   • Duplicate full copies of optimizer state, gradients, and parameters on every GPU
   • Duplicate copies of the training data on each GPU
   • Redundant network cables between nodes
   • Repeated forward passes over the same batch
2. Why is optimizer state often the biggest memory hog with Adam?
   • Adam stores no extra state at all
   • Adam keeps additional per-parameter values (like momentum and variance), often several times the model size
   • Adam duplicates the entire dataset
   • Adam requires storing every past gradient forever
3. What does ZeRO Stage 3 shard that Stages 1 and 2 do not?
   • The training data
   • The loss function
   • The model parameters themselves
   • The learning rate schedule
4. In ZeRO Stage 3, how does a GPU get the full parameters it needs for a layer's forward pass?
   • It never needs the full parameters
   • It downloads them from the internet
   • It recomputes them from scratch each time
   • It performs an all-gather to collect the missing slices, then discards them afterward
5. Which PyTorch feature natively implements ZeRO-style sharding?
   • FSDP (Fully Sharded Data Parallel)
   • DataLoader
   • TorchScript
   • Autograd
6. What is the core tradeoff ZeRO makes to save memory?
   • It sacrifices model accuracy for speed
   • It adds extra communication to gather and scatter shards as needed
   • It removes layers from the model
   • It only works with very small models
7. Which state is partitioned in ZeRO Stage 1?
   • Only parameters, with optimizer states replicated
   • Parameters, gradients, and optimizer states all sharded
   • Optimizer states, while parameters and gradients remain replicated
   • Only input activations and labels
8. What is a practical cost of ZeRO-style sharding?
   • It removes the need to calculate gradients
   • It makes every optimizer update local with no data exchange
   • It requires the model to be retrained from scratch for every step
   • Additional communication to gather parameters and distribute gradients