Model and Pipeline Parallelism Quiz

Answers and explanations appear as you take the quiz.

1. What fundamental problem do model and pipeline parallelism solve?
   • A model being too large to fit in a single GPU's memory
   • Slow data loading from disk
   • Gradients exploding during training
   • Too few training examples available
2. How does tensor (intra-layer) parallelism split work?
   • It assigns whole layers to different GPUs
   • It splits the math inside a layer, like a matrix multiply, across GPUs
   • It duplicates each layer on every GPU
   • It runs each layer twice for redundancy
3. What is the 'bubble' in naive pipeline parallelism?
   • A memory leak that grows over time
   • A duplicate copy of the optimizer state
   • Idle time when downstream GPUs wait for upstream stages to produce activations
   • A spike in gradient values
4. How does pipeline parallelism reduce the bubble?
   • By using a smaller learning rate
   • By removing some layers from the model
   • By storing activations on the CPU
   • By splitting each batch into micro-batches that flow through stages in a staggered way
5. Why is tensor parallelism typically kept within a single node?
   • Because its frequent all-reduce of partial results benefits from fast intra-node links like NVLink
   • Because it does not require any communication
   • Because nodes cannot share data at all
   • Because it only supports one GPU
6. Which framework is best known for tensor parallelism in transformer training?
   • scikit-learn
   • NVIDIA Megatron-LM
   • OpenCV
   • Pandas
7. What data must move between stages in pipeline parallelism during a forward pass?
   • The entire training dataset after each layer
   • Only the optimizer’s learning rate
   • Activation tensors produced by one stage and consumed by the next
   • No data, because stages are independent models
8. Why does tensor parallelism often require fast communication within its group?
   • It assigns each GPU a separate dataset and never synchronizes
   • It communicates only once when the model is loaded
   • It uses no cross-device operations
   • Devices exchange partial results or activations within layers during computation