DeepSpeed ak Megatron

Sooy def quiz bi, tontu yi ak leeral yi dañuy feeñ.

1. Lan mooy jubluwaay bu njëkk ci optimisatëru ZeRO bu DeepSpeed?
   • Ngir xaaj etaa yi, degrade yi ak parametre yi ci GPU yi ngir denc mémoire bi
   • Ngir kompresse model bi mujj ngir dugal ko
   • Tekki ay misaal ci diggante kaadar yi
   • Ngir etiketu ci saasi done tàggat
2. Paralelism tensor, ni ñu ko njëkka amal ci Megatron-LM, naka lay xaajalee model bi?
   • Soo defee ay layer yu wuute ci GPU yu wuute
   • Ni ñuy daggee benn-benn matrix ci biir benn layer ci GPU yi
   • Soo joxee GPU bu nekk ay done yu wuute
   • Ni ñuy denc ay jéego yu wuute ci GPU yu wuute
3. Paralelismu tuyo wuute na ak paralelismu tensor ndax dafay xaaj model bi ci:
   • Dagg matris ci biir couche bu nekk
   • Ñaareelu model bi yépp ci GPU bu nekk
   • Jox ay layer (stage) yu wuute ay GPU yu wuute
   • Diisaay yu bari ngir wàññi njub
4. Ban etape ZeRO mooy joxe sakkanal mémoire bu gëna mag ci sharding model parametre yi ci seen bopp?
   • etape 1
   • 2ème étape
   • etape 0
   • 3ème étape
5. Lan mooy 'checkpointing' ngir sakkanal mémoire bi ci diiru tàggat yaram?
   • Extra calcul, ci xaymawaat aktivaasioŋ yi ci diir bi ñuy delloo ci ginaaw
   • model bu jaar yoon
   • Yaatuwaayu band reso bi
   • Limu diisaay yi
6. Lan moo waral ñuy woowe boole pexe yooyu 'parallelism 3D'?
   • Ci kàrtu grafik 3D kese lay dox
   • Dafay boole ay done, tensor, ak paralelismu tuyo ci ñatti axe
   • Dafay jëfandikoo ñetti GPU yu gëna bari
   • Dafay joxe ay model ci ñatti dimension
7. Which ZeRO stage first shards optimizer state across data-parallel workers?
   • etape 0
   • 2ème étape
   • etape 1
   • 3ème étape
8. What does CPU or NVMe offloading do in large-model training stacks?
   • Copies the full model to every GPU
   • Removes all inter-GPU communication
   • Changes tensor parallelism into reinforcement learning
   • Moves some model or optimizer state off GPU memory to increase the available GPU-memory budget