Sequence Parallelism and Ring Attention Quiz

Answers and explanations appear as you take the quiz.

1. Along which dimension does sequence parallelism split the data?
   • The hidden/feature dimension
   • The token (sequence) dimension
   • The batch dimension only
   • The number of layers
2. What does Ring Attention pass between GPUs arranged in a ring?
   • Full model weights
   • Optimizer states
   • Final logits
   • Key/value blocks
3. How can Ring Attention combine partial attention blocks without approximating the full attention pattern?
   • Online softmax with a running maximum and normalizer
   • Randomly dropping unseen key blocks
   • Truncating every distant key
   • Replacing all attention scores with zeros
4. Why doesn't Ring Attention require any single GPU to store the full K/V?
   • It discards most keys
   • It uses a smaller model
   • K/V is streamed in blocks and only partial results are accumulated
   • Attention is approximated, not exact
5. When can Ring Attention reduce the exposed time spent transferring K/V blocks?
   • Whenever the model has any number of parameters
   • Only when every device stores the full sequence
   • By eliminating all transfers between devices
   • When useful block computation overlaps those transfers
6. What complication does causal masking introduce for Ring Attention load balancing?
   • It makes the model non-differentiable
   • Later tokens attend to more keys, causing uneven work per GPU
   • It removes the need for a ring
   • It doubles the parameter count
7. A new Ring Attention block has larger attention scores than previous blocks. Why rescale the earlier softmax sums?
   • To discard the earlier keys permanently
   • To replace all token embeddings
   • To keep accumulated terms consistent with the updated running maximum
   • To reduce the vocabulary size
8. Under causal masking with contiguous sequence shards, which tokens can require more attention work?
   • Later tokens that can attend to more preceding positions
   • All tokens require exactly one key
   • Only padding tokens at the start
   • Earlier tokens because they can see the future