paralelism tensor ngir model yu mag

Sooy def quiz bi, tontu yi ak leeral yi dañuy feeñ.

1. Lan mooy xaaj paralelism tensor ci GPU yi?
   • matrix yu diis yi ci biir benn couche
   • Layers yépp ñu ngi leen di def ci GPU yu wuute
   • ay done yu wuute ci tàggat yaram
   • Tolluwaayu jàng bu optimisatër bi
2. Ci xaajalug MLP bu nuroo ak Megatron, naka lañuy xaajalee ñaari matris yu diis yi ngir wàññi jokkoo bi?
   • Ñoom ñaar ci kolon
   • Kolon bu njëkk bi, ñaareel bi
   • Ñoom ñaar ñépp
   • Duñu leen xaaj dara
3. Lu tax ñuy faral di denc paralelism tensor ci benn node?
   • Node yi mënoo doxal GPU yi
   • GPUs ci node yu wuute dañuy jëfandikoo math yu wuute
   • Dafa soxla jokkoo bu bari bandwidth bu bari te NVLink moo ko gëna mëna jàppale
   • Yoon làaje la
4. naka lañuy méngale mekaniismu bàyyi xel ci suufu paralelism tensor?
   • Soo joxee GPU bu nekk xel mi yépp ci xayma bi
   • Ni ngay bàyyi xel ci lépp
   • Ni ñuy soppi bàyyi xel ci benn convolution
   • Buñu xaajalee ci boppu yi, GPU bu nekk dafay jëfandikoo benn subset
5. Ban jëf bu mbooloo mooy faral di boole ay resultaa yu xaaj ci paralelism tensor?
   • Wàññi lépp
   • ab kopi fichier
   • Jokkoo ci base de done
   • Seetug DNS
6. naka la paralelism tensoor di méngoo ak 'paralelism 3D'?
   • Dafay wecci yeneen xeeti paralelism yépp
   • Dafa boole ak gasoduc ak paralelism done
   • Dafay dox rek te amul beneen paralelism
   • Dañu koy jëfandikoo ci modelu nataal kese
7. After the row-wise partition of an MLP’s second matrix, why must GPUs combine partial outputs?
   • The GPUs each hold a different optimizer
   • The result is a vocabulary probability that must be sorted
   • Each GPU computed a partial contribution to the same output, so the pieces must be summed
   • The model needs to average separate datasets
8. Why can interconnect latency limit tensor-parallel scaling across distant nodes?
   • Tensor parallelism never communicates between GPUs
   • The attention mask must be copied after every token
   • Every GPU trains a separate model
   • Collectives are required repeatedly within layers, so communication latency accumulates