Idanwo ti o sopọ mọ itọsọna · Alabọde Ipele

Data Parallelism adanwo

Review replicated models, gradient synchronization, batch sizing, communication costs, and the limits of scaling training across workers.

Jẹmọ itọsọna awọn ọnaData Parallelism
Ibeere 1 ti 8

Ni afiwe data boṣewa, kini GPU kọọkan mu?

Awotẹlẹ gbogbo ibeere8 ibeere

Awọn idahun ati awọn alaye han bi o ṣe n mu ibeere naa.

  1. Ni afiwe data boṣewa, kini GPU kọọkan mu?
    • Ẹda pipe ti awoṣe, ṣiṣe bibẹ pẹlẹbẹ ti o yatọ ti ipele naa
    • Bibẹ pẹlẹbẹ ti o yatọ ti awọn fẹlẹfẹlẹ awoṣe
    • Nikan ipo iṣapeye, pẹlu awọn iwuwo ti o fipamọ ni aarin
    • Ẹya fisinuirindigbindigbin ti awoṣe pẹlu awọn paramita diẹ
  2. Iṣiṣẹ ibaraẹnisọrọ wo ni o tọju awọn ẹda awoṣe ni mimuṣiṣẹpọ ni igbesẹ kọọkan?
    • Broadcast, eyiti o daakọ data GPU kan si gbogbo awọn miiran
    • Gbogbo-din, eyiti o ṣe aropin awọn iwọn gradients kọja gbogbo awọn GPUs
    • Tuka, eyiti o pin data laarin awọn GPU
    • Ko si ibaraẹnisọrọ ti wa ni ti nilo laarin GPUs
  3. What memory limit remains in plain data parallelism?
    • The full model is sharded automatically across the GPUs
    • Only the input batch uses device memory
    • Each GPU keeps model parameters and optimizer state replicated, limiting the model size that fits
    • Each worker stores different model layers by default
  4. Which factors does NCCL use when selecting collective algorithms?
    • Only the random seed and learning rate
    • Only the total number of model parameters
    • The optimizer type and validation score
    • Message size and topology
  5. Bawo ni PyTorch DistributedDataParallel ṣe tọju lairi ibaraẹnisọrọ bi?
    • Nipa agbekọja gradient gbogbo-din pẹlu iwọle sẹhin ti nlọ lọwọ
    • Nípa sísẹ́ ìsiṣẹ́pọ̀ dídíẹ̀tì lórí àwọn ìṣísẹ̀ yípo
    • Nipa titẹ gbogbo awọn gradients si nọmba kan
    • Nipa ṣiṣe siwaju kọja lori Sipiyu
  6. With equal local batch size B on each of W data-parallel workers, what is the global batch for one synchronized step?
    • B divided by W
    • B multiplied by W, before any gradient-accumulation factor
    • Always exactly W samples
    • The model parameter count divided by B
  7. Why may adding more data-parallel GPUs fail to increase throughput proportionally?
    • All-reduce makes model parameters disappear
    • The model is copied only to one worker
    • Gradient communication, data loading, and synchronization add overhead
    • More workers always reduce the global batch to zero
  8. What does gradient accumulation let a data-parallel training loop do?
    • Shard parameters automatically across all workers
    • Skip forward and backward computation
    • Replace the model with an all-reduce operation
    • Combine gradients from multiple microbatches before an optimizer update