数据并行性测验

当您参加测验时，就会出现答案和解释。

1. 在标准数据并行性中，每个 GPU 保存什么？
   • 模型的完整副本，处理批次的不同部分
   • 模型层的不同切片
   • 仅优化器状态，权重集中存储
   • 模型的压缩版本，参数较少
2. 哪个通信操作使模型副本每一步保持同步？
   • 广播，将一个 GPU 的数据复制到所有其他 GPU
   • All-reduce，对所有 GPU 上的梯度进行平均
   • Scatter，在 GPU 之间划分数据
   • GPU之间不需要通信
3. What memory limit remains in plain data parallelism?
   • The full model is sharded automatically across the GPUs
   • Only the input batch uses device memory
   • Each GPU keeps model parameters and optimizer state replicated, limiting the model size that fits
   • Each worker stores different model layers by default
4. Which factors does NCCL use when selecting collective algorithms?
   • Only the random seed and learning rate
   • Only the total number of model parameters
   • The optimizer type and validation score
   • Message size and topology
5. PyTorch DistributedDataParallel 如何隐藏通信延迟？
   • 通过重叠梯度全归约与正在进行的向后传递
   • 通过在交替步骤上跳过梯度同步
   • 通过将所有梯度压缩为一个数字
   • 通过在 CPU 上运行前向传播
6. With equal local batch size B on each of W data-parallel workers, what is the global batch for one synchronized step?
   • B divided by W
   • B multiplied by W, before any gradient-accumulation factor
   • Always exactly W samples
   • The model parameter count divided by B
7. Why may adding more data-parallel GPUs fail to increase throughput proportionally?
   • All-reduce makes model parameters disappear
   • The model is copied only to one worker
   • Gradient communication, data loading, and synchronization add overhead
   • More workers always reduce the global batch to zero
8. What does gradient accumulation let a data-parallel training loop do?
   • Shard parameters automatically across all workers
   • Skip forward and backward computation
   • Replace the model with an all-reduce operation
   • Combine gradients from multiple microbatches before an optimizer update