Tensor Parallelism for Large Models Quiz

Answers and explanations appear as you take the quiz.

1. What does tensor parallelism split across GPUs?
   • The weight matrices within a single layer
   • Entire layers placed on separate GPUs
   • Different training datasets
   • The optimizer learning rate
2. In the Megatron-style MLP split, how are the two weight matrices partitioned to minimize communication?
   • Both column-wise
   • First column-wise, second row-wise
   • Both row-wise
   • They are not partitioned at all
3. Why is tensor parallelism usually kept within a single node?
   • Nodes cannot run GPUs
   • GPUs in different nodes use different math
   • It needs frequent high-bandwidth communication best served by NVLink
   • It is a legal requirement
4. How is the attention mechanism typically parallelized under tensor parallelism?
   • By giving every GPU the full attention computation redundantly
   • By skipping attention entirely
   • By converting attention to a convolution
   • By splitting across attention heads, each GPU handling a subset
5. What collective operation commonly combines partial results in tensor parallelism?
   • An all-reduce
   • A file copy
   • A database join
   • A DNS lookup
6. How does tensor parallelism typically fit into '3D parallelism'?
   • It replaces all other forms of parallelism
   • It is combined with pipeline and data parallelism
   • It only works without any other parallelism
   • It is used solely for image models
7. After the row-wise partition of an MLP’s second matrix, why must GPUs combine partial outputs?
   • The GPUs each hold a different optimizer
   • The result is a vocabulary probability that must be sorted
   • Each GPU computed a partial contribution to the same output, so the pieces must be summed
   • The model needs to average separate datasets
8. Why can interconnect latency limit tensor-parallel scaling across distant nodes?
   • Tensor parallelism never communicates between GPUs
   • The attention mask must be copied after every token
   • Every GPU trains a separate model
   • Collectives are required repeatedly within layers, so communication latency accumulates