Idanwo ti o sopọ mọ itọsọna · Lile Ipele

DeepSpeed ati Megatron Training Stacks Quiz

Checks ZeRO sharding stages, tensor and pipeline parallelism, activation checkpointing, offloading and combined 3D parallel training.

Jẹmọ itọsọna awọn ọnaDeepspeed Ati Megatron Ikẹkọ Awọn akopọ
Ibeere 1 ti 8

Kini idi akọkọ ti DeepSpeed's ZeRO optimizer?

Awotẹlẹ gbogbo ibeere8 ibeere

Awọn idahun ati awọn alaye han bi o ṣe n mu ibeere naa.

  1. Kini idi akọkọ ti DeepSpeed's ZeRO optimizer?
    • Si awọn ipinlẹ iṣapeye shard, awọn gradients, ati awọn paramita kọja awọn GPU lati fi iranti pamọ
    • Lati compress ik awoṣe fun imuṣiṣẹ
    • Lati tumọ awọn awoṣe laarin awọn ilana
    • Lati ṣe aami data ikẹkọ laifọwọyi
  2. Tensor parallelism, bi aṣáájú-ọnà ni Megatron-LM, pin awoṣe bi?
    • Nipa fifi awọn ipele oriṣiriṣi sori awọn GPU oriṣiriṣi
    • Nipa dida awọn isodipupo matrix kọọkan laarin ipele kan kọja awọn GPUs
    • Nipa fifun kọọkan GPU kan ti o yatọ ipele ti data
    • Nipa titoju awọn igbesẹ akoko oriṣiriṣi lori awọn GPU oriṣiriṣi
  3. Ibaṣepọ pipeline yato si afiwera tensor nitori pe o pin awoṣe nipasẹ:
    • Awọn matrices gige ni inu Layer kọọkan
    • Duplicating gbogbo awoṣe lori gbogbo GPU
    • Yiyan awọn ipele oriṣiriṣi (awọn ipele) si awọn GPU oriṣiriṣi
    • Pipin awọn iwuwo si iwọn konge
  4. Ipele ZeRO wo ni o pese awọn ifowopamọ iranti ti o tobi julọ nipa tun pin awọn paramita awoṣe funrararẹ?
    • Ipele 1
    • Ipele 2
    • Ipele 0
    • Ipele 3
  5. Kini 'iṣayẹwo imuṣiṣẹ' ṣe iṣowo kuro lati fi iranti pamọ lakoko ikẹkọ?
    • Iṣiro afikun, nipa ṣiṣe atunṣiro awọn iṣẹ ṣiṣe nigba iwọle sẹhin
    • Awoṣe deede
    • Bandiwidi nẹtiwọki
    • Awọn nọmba ti fẹlẹfẹlẹ
  6. Kini idi ti apapọ awọn ilana wọnyi nigbagbogbo n pe ni 'parallelism 3D'?
    • O ṣiṣẹ nikan lori awọn kaadi eya 3D
    • O daapọ data, tensor, ati pipeline parallelism pẹlu awọn aake mẹta
    • O nlo awọn GPU mẹta ti o pọju
    • O ṣe awọn awoṣe ni awọn iwọn mẹta
  7. Which ZeRO stage first shards optimizer state across data-parallel workers?
    • Ipele 0
    • Ipele 2
    • Ipele 1
    • Ipele 3
  8. What does CPU or NVMe offloading do in large-model training stacks?
    • Copies the full model to every GPU
    • Removes all inter-GPU communication
    • Changes tensor parallelism into reinforcement learning
    • Moves some model or optimizer state off GPU memory to increase the available GPU-memory budget