序列并行性和环注意力测验

当您参加测验时，就会出现答案和解释。

1. 序列并行沿着哪个维度分割数据？
   • 隐藏/特征维度
   • 令牌（序列）维度
   • 仅批次尺寸
   • 层数
2. 环形注意力机制在排列成环形的 GPU 之间传递什么？
   • 全模型重量
   • 优化器状态
   • 最终逻辑
   • 键/值块
3. How can Ring Attention combine partial attention blocks without approximating the full attention pattern?
   • Online softmax with a running maximum and normalizer
   • Randomly dropping unseen key blocks
   • Truncating every distant key
   • Replacing all attention scores with zeros
4. 为什么 Ring Attention 不需要任何单个 GPU 来存储完整的 K/V？
   • 它会丢弃大多数键
   • 它使用较小的模型
   • K/V 以块的形式传输，仅累积部分结果
   • 注意力是近似的，而不是精确的
5. When can Ring Attention reduce the exposed time spent transferring K/V blocks?
   • Whenever the model has any number of parameters
   • Only when every device stores the full sequence
   • By eliminating all transfers between devices
   • When useful block computation overlaps those transfers
6. 因果屏蔽会给环注意力负载均衡带来什么复杂性？
   • 它使模型不可微分
   • 后来的令牌处理更多的键，导致每个 GPU 的工作不均匀
   • 它消除了对戒指的需要
   • 参数数量加倍
7. A new Ring Attention block has larger attention scores than previous blocks. Why rescale the earlier softmax sums?
   • To discard the earlier keys permanently
   • To replace all token embeddings
   • To keep accumulated terms consistent with the updated running maximum
   • 减少词汇量
8. Under causal masking with contiguous sequence shards, which tokens can require more attention work?
   • Later tokens that can attend to more preceding positions
   • All tokens require exactly one key
   • Only padding tokens at the start
   • Earlier tokens because they can see the future