Guide-yakabatana mibvunzo · Zvakaoma Nhanho

Muenzaniso uye Pipeline Parallelism Quiz

Explains tensor and pipeline partitioning, activation transfers, microbatch schedules, pipeline bubbles, and communication tradeoffs.

Nzira dzekutungamira dzinoenderanaModel Uye Pipeline Parallelism
Mubvunzo 1 ye 8

Nderipi dambudziko rakakosha rinogadziriswa modhi uye pombi parallelism?

Tarisa mubvunzo wese8 mibvunzo

Mhinduro uye tsananguro zvinoonekwa paunenge uchitora bvunzo.

  1. Nderipi dambudziko rakakosha rinogadziriswa modhi uye pombi parallelism?
    • Modhi yakakurisa kuti ikwane mundangariro imwe chete yeGPU
    • Inononoka kurodha data kubva kudhisiki
    • Gradients kuputika panguva yekudzidziswa
    • Mienzaniso mishoma yekudzidzisa iripo
  2. Ko tensor (intra-layer) parallelism inopatsanura sei?
    • Inopa zvikamu zvese kune akasiyana maGPU
    • Iyo inotsemura masvomhu mukati mechikamu, senge matrix kuwanda, mhiri kweGPUs
    • Inodzokorora dhizaini yega yega paGPU yega yega
    • Inomhanya imwe neimwe layer kaviri kune redundancy
  3. Chii chinonzi 'bubble' mune naive pipeline parallelism?
    • Memory leak inokura nekufamba kwenguva
    • Iyo yakapetwa kopi yeiyo optimizer state
    • Idle nguva apo yakadzika maGPU anomirira nhanho dzekumusoro kuti dzigadzire activation
    • A spike in gradient values
  4. Kufanana kwepombi kunoderedza sei bubble?
    • Nokushandisa mwero muduku wokudzidza
    • Nekubvisa mamwe matinji kubva mumuenzaniso
    • Nekuchengetedza ma activation paCPU
    • Nekukamura batch yega yega kuita madiki-mabhechi anoyerera nematanho nenzira yakadzama
  5. Sei tensor parallelism ichiwanzo chengetwa mukati meimwe node?
    • Nekuti iyo inowanzoitika-yese-kudzikisa kwechikamu mhedzisiro inobatsira kubva nekukurumidza intra-node zvinongedzo seNVLink
    • Nokuti hazvidi chero kutaurirana
    • Nekuti node haigone kugovera data zvachose
    • Nekuti inotsigira GPU imwe chete
  6. Ndechipi chimiro chinozivikanwa zvakanyanya nezve tensor parallelism mu transformer kudzidzisa?
    • scikit-dzidza
    • NVIDIA Megatron-LM
    • OpenCV
    • Pandas
  7. What data must move between stages in pipeline parallelism during a forward pass?
    • The entire training dataset after each layer
    • Only the optimizer’s learning rate
    • Activation tensors produced by one stage and consumed by the next
    • No data, because stages are independent models
  8. Why does tensor parallelism often require fast communication within its group?
    • It assigns each GPU a separate dataset and never synchronizes
    • It communicates only once when the model is loaded
    • It uses no cross-device operations
    • Devices exchange partial results or activations within layers during computation