Idanwo ti o sopọ mọ itọsọna · Lile Ipele

ZeRO ati Sharded Optimizers adanwo

Covers ZeRO stages, Adam state memory, parameter gathering, gradient reduction, offload options, and communication costs.

Jẹmọ itọsọna awọn ọnaOdo Ati Sharded Optimizers
Ibeere 1 ti 8

Apọju wo ni ZeRO yọkuro ni akawe si afiwera data itele?

Awotẹlẹ gbogbo ibeere8 ibeere

Awọn idahun ati awọn alaye han bi o ṣe n mu ibeere naa.

  1. Apọju wo ni ZeRO yọkuro ni akawe si afiwera data itele?
    • Ṣe pidánpidán ni kikun awọn adakọ ti ipo iṣapeye, awọn gradients, ati awọn paramita lori gbogbo GPU
    • Awọn ẹda ẹda ti data ikẹkọ lori GPU kọọkan
    • Awọn kebulu nẹtiwọọki laiṣe laarin awọn apa
    • Tun siwaju kọja lori ipele kanna
  2. Kini idi ti ipo optimizer nigbagbogbo jẹ hog iranti nla julọ pẹlu Adam?
    • Adam tọjú ko si afikun ipinle ni gbogbo
    • Adam tọju awọn iye afikun-paramita (bii ipa ati iyatọ), nigbagbogbo ni igba pupọ iwọn awoṣe
    • Adam pidánpidán gbogbo dataset
    • Adam nilo fifipamọ gbogbo gradient ti o kọja lailai
  3. Kini Shard Ipele 3 ZeRO ti Awọn ipele 1 ati 2 ko ṣe?
    • Awọn data ikẹkọ
    • Iṣẹ isonu
    • Awọn paramita awoṣe ara wọn
    • Iṣeto oṣuwọn ẹkọ
  4. Ni Ipele ZeRO 3, bawo ni GPU ṣe gba awọn ayeraye ni kikun ti o nilo fun iwọle siwaju Layer kan?
    • Ko nilo awọn paramita kikun
    • O gba wọn lati ayelujara
    • O recomputes wọn lati ibere kọọkan akoko
    • O ṣe apejọ gbogbo lati gba awọn ege ti o padanu, lẹhinna sọ wọn nù lẹhin naa
  5. Ẹya PyTorch wo ni o ṣe imuse aṣa Sharding ZeRO?
    • FSDP (Data Pipin Ni kikun)
    • Loader Data
    • ÒgùṣọScript
    • Autograd
  6. Kini iṣowo mojuto ZeRO ṣe lati fi iranti pamọ?
    • O rubọ deede awoṣe fun iyara
    • O ṣe afikun ibaraẹnisọrọ afikun lati ṣajọ ati tuka awọn ọpa bi o ti nilo
    • O yọ awọn fẹlẹfẹlẹ lati awoṣe
    • O ṣiṣẹ nikan pẹlu awọn awoṣe kekere pupọ
  7. Which state is partitioned in ZeRO Stage 1?
    • Only parameters, with optimizer states replicated
    • Parameters, gradients, and optimizer states all sharded
    • Optimizer states, while parameters and gradients remain replicated
    • Only input activations and labels
  8. What is a practical cost of ZeRO-style sharding?
    • It removes the need to calculate gradients
    • It makes every optimizer update local with no data exchange
    • It requires the model to be retrained from scratch for every step
    • Additional communication to gather parameters and distribute gradients