Ikibazo gihuza ibibazo · Hagati Urwego
Ikibazo cyo Kuringaniza Ikibazo
Ongera usuzume icyitegererezo cyakoporowe, guhuza buhoro buhoro, ingano yicyiciro, ikiguzi cyitumanaho, nimbibi zamahugurwa yo gupima abakozi.
Ikibazo 1 Bya 8
Mubisanzwe amakuru abangikanye, buri GPU ifata iki?
Reba buri kibazo8 ibibazo
Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.
- Mubisanzwe amakuru abangikanye, buri GPU ifata iki?
- Kopi yuzuye yicyitegererezo, itunganya ibice bitandukanye byicyiciro
- Igice gitandukanye cyurugero rwicyitegererezo
- Gusa optimizer leta, hamwe nuburemere bubitswe hagati
- Impapuro zifunitse zicyitegererezo hamwe nibintu bike
- Ni ubuhe buryo bwo gutumanaho butuma icyitegererezo gikoporora buri ntambwe?
- Broadcast, ikoporora amakuru ya GPU kubandi bose
- Byose-bigabanye, impuzandengo ya gradients muri GPU zose
- Ikwirakwiza, igabanya amakuru muri GPUs
- Nta tumanaho rikenewe hagati ya GPUs
- What memory limit remains in plain data parallelism?
- The full model is sharded automatically across the GPUs
- Only the input batch uses device memory
- Each GPU keeps model parameters and optimizer state replicated, limiting the model size that fits
- Each worker stores different model layers by default
- Which factors does NCCL use when selecting collective algorithms?
- Only the random seed and learning rate
- Only the total number of model parameters
- The optimizer type and validation score
- Message size and topology
- Nigute PyTorch IkwirakwizwaDataParllel ihisha itinda ryitumanaho?
- Mugutwikiriye gradient byose-kugabanya hamwe ninyuma yinyuma yinyuma
- Mugusiba icyiciro cya syncronisation ku ntambwe zisimburana
- Mugukanda gradients zose kumubare umwe
- Mugukoresha inzira yimbere kuri CPU
- With equal local batch size B on each of W data-parallel workers, what is the global batch for one synchronized step?
- B divided by W
- B multiplied by W, before any gradient-accumulation factor
- Always exactly W samples
- The model parameter count divided by B
- Why may adding more data-parallel GPUs fail to increase throughput proportionally?
- All-reduce makes model parameters disappear
- The model is copied only to one worker
- Gradient communication, data loading, and synchronization add overhead
- More workers always reduce the global batch to zero
- What does gradient accumulation let a data-parallel training loop do?
- Shard parameters automatically across all workers
- Skip forward and backward computation
- Replace the model with an all-reduce operation
- Combine gradients from multiple microbatches before an optimizer update