Ikibazo gihuza ibibazo · Hagati Urwego

Ikibazo cyo Kuringaniza Ikibazo

Ongera usuzume icyitegererezo cyakoporowe, guhuza buhoro buhoro, ingano yicyiciro, ikiguzi cyitumanaho, nimbibi zamahugurwa yo gupima abakozi.

Inzira zijyanye no kuyoboraKuringaniza amakuru
Ikibazo 1 Bya 8

Mubisanzwe amakuru abangikanye, buri GPU ifata iki?

Reba buri kibazo8 ibibazo

Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.

  1. Mubisanzwe amakuru abangikanye, buri GPU ifata iki?
    • Kopi yuzuye yicyitegererezo, itunganya ibice bitandukanye byicyiciro
    • Igice gitandukanye cyurugero rwicyitegererezo
    • Gusa optimizer leta, hamwe nuburemere bubitswe hagati
    • Impapuro zifunitse zicyitegererezo hamwe nibintu bike
  2. Ni ubuhe buryo bwo gutumanaho butuma icyitegererezo gikoporora buri ntambwe?
    • Broadcast, ikoporora amakuru ya GPU kubandi bose
    • Byose-bigabanye, impuzandengo ya gradients muri GPU zose
    • Ikwirakwiza, igabanya amakuru muri GPUs
    • Nta tumanaho rikenewe hagati ya GPUs
  3. What memory limit remains in plain data parallelism?
    • The full model is sharded automatically across the GPUs
    • Only the input batch uses device memory
    • Each GPU keeps model parameters and optimizer state replicated, limiting the model size that fits
    • Each worker stores different model layers by default
  4. Which factors does NCCL use when selecting collective algorithms?
    • Only the random seed and learning rate
    • Only the total number of model parameters
    • The optimizer type and validation score
    • Message size and topology
  5. Nigute PyTorch IkwirakwizwaDataParllel ihisha itinda ryitumanaho?
    • Mugutwikiriye gradient byose-kugabanya hamwe ninyuma yinyuma yinyuma
    • Mugusiba icyiciro cya syncronisation ku ntambwe zisimburana
    • Mugukanda gradients zose kumubare umwe
    • Mugukoresha inzira yimbere kuri CPU
  6. With equal local batch size B on each of W data-parallel workers, what is the global batch for one synchronized step?
    • B divided by W
    • B multiplied by W, before any gradient-accumulation factor
    • Always exactly W samples
    • The model parameter count divided by B
  7. Why may adding more data-parallel GPUs fail to increase throughput proportionally?
    • All-reduce makes model parameters disappear
    • The model is copied only to one worker
    • Gradient communication, data loading, and synchronization add overhead
    • More workers always reduce the global batch to zero
  8. What does gradient accumulation let a data-parallel training loop do?
    • Shard parameters automatically across all workers
    • Skip forward and backward computation
    • Replace the model with an all-reduce operation
    • Combine gradients from multiple microbatches before an optimizer update