DeepSpeed and Megatron Training Stacks Quiz

Answers and explanations appear as you take the quiz.

1. What is the primary purpose of DeepSpeed's ZeRO optimizer?
   • To shard optimizer states, gradients, and parameters across GPUs to save memory
   • To compress the final model for deployment
   • To translate models between frameworks
   • To automatically label training data
2. Tensor parallelism, as pioneered in Megatron-LM, splits the model how?
   • By putting different layers on different GPUs
   • By slicing individual matrix multiplications within a layer across GPUs
   • By giving each GPU a different batch of data
   • By storing different time steps on different GPUs
3. Pipeline parallelism differs from tensor parallelism because it splits the model by:
   • Slicing matrices inside each layer
   • Duplicating the whole model on every GPU
   • Assigning different layers (stages) to different GPUs
   • Quantizing weights to lower precision
4. Which ZeRO stage provides the greatest memory savings by also sharding the model parameters themselves?
   • Stage 1
   • Stage 2
   • Stage 0
   • Stage 3
5. What does 'activation checkpointing' trade away to save memory during training?
   • Extra compute, by recomputing activations during the backward pass
   • Model accuracy
   • Network bandwidth
   • The number of layers
6. Why is combining these techniques often called '3D parallelism'?
   • It only works on 3D graphics cards
   • It combines data, tensor, and pipeline parallelism along three axes
   • It uses three GPUs maximum
   • It renders models in three dimensions
7. Which ZeRO stage first shards optimizer state across data-parallel workers?
   • Stage 0
   • Stage 2
   • Stage 1
   • Stage 3
8. What does CPU or NVMe offloading do in large-model training stacks?
   • Copies the full model to every GPU
   • Removes all inter-GPU communication
   • Changes tensor parallelism into reinforcement learning
   • Moves some model or optimizer state off GPU memory to increase the available GPU-memory budget