Idanwo ti o sopọ mọ itọsọna · Lile Ipele

Tensor Parallelism fun Idanwo Awọn awoṣe Nla

Checks how tensor parallelism shards layer matrices, combines partial outputs, splits attention heads and fits into multi-axis parallelism.

Jẹmọ itọsọna awọn ọnaTensor Parallelism Fun Awọn awoṣe Nla
Ibeere 1 ti 8

Kini parallelism tensor pin kọja awọn GPU?

Awotẹlẹ gbogbo ibeere8 ibeere

Awọn idahun ati awọn alaye han bi o ṣe n mu ibeere naa.

  1. Kini parallelism tensor pin kọja awọn GPU?
    • Awọn iwuwo matrices laarin kan nikan Layer
    • Gbogbo awọn ipele ti a gbe sori awọn GPU lọtọ
    • O yatọ si ikẹkọ datasets
    • Oṣuwọn ẹkọ ti o dara julọ
  2. Ni Megatron-ara MLP pipin, bawo ni a ṣe pin awọn matiri iwuwo meji lati dinku ibaraẹnisọrọ?
    • Mejeeji ọwọn-ọlọgbọn
    • Ọgbọn ọwọn akọkọ, ọgbọn-ila keji
    • Mejeeji kana-ọlọgbọn
    • Wọn ko pin rara
  3. Kini idi ti ifaramọ tensor maa n wa laarin ipade kan?
    • Awọn apa ko le ṣiṣe awọn GPUs
    • GPUs ni orisirisi awọn apa lo o yatọ si isiro
    • O nilo ibaraẹnisọrọ bandiwidi giga loorekoore iṣẹ ti o dara julọ nipasẹ NVLink
    • O jẹ ibeere labẹ ofin
  4. Bawo ni ẹrọ ifarabalẹ ṣe deede ni afiwe labẹ isọdọkan tensor?
    • Nipa fifun gbogbo GPU ni iṣiro akiyesi ni kikun laiṣe
    • Nipa fo akiyesi patapata
    • Nipa yiyipada akiyesi si a convolution
    • Nipa pipin kọja awọn olori akiyesi, GPU kọọkan n ṣakoso ipin kan
  5. Iṣiṣẹ apapọ wo ni o ṣajọpọ awọn abajade apa kan ni afiwe tensor?
    • An gbogbo-din
    • Ẹda faili kan
    • A da database
    • Ayẹwo DNS kan
  6. Bawo ni parallelism tensor ṣe deede ni ibamu si 'parallelism 3D'?
    • O rọpo gbogbo awọn ọna miiran ti parallelism
    • O ti wa ni idapo pelu opo gigun ti epo ati data parallelism
    • O ṣiṣẹ nikan laisi afiwera miiran
    • O ti lo fun awọn awoṣe aworan nikan
  7. After the row-wise partition of an MLP’s second matrix, why must GPUs combine partial outputs?
    • The GPUs each hold a different optimizer
    • The result is a vocabulary probability that must be sorted
    • Each GPU computed a partial contribution to the same output, so the pieces must be summed
    • The model needs to average separate datasets
  8. Why can interconnect latency limit tensor-parallel scaling across distant nodes?
    • Tensor parallelism never communicates between GPUs
    • The attention mask must be copied after every token
    • Every GPU trains a separate model
    • Collectives are required repeatedly within layers, so communication latency accumulates