Kedis ku xidhan hagaha · Adag Heerka

Kediska ZeRO iyo Sharded Optimizers

Waxay dabooshaa marxaladaha ZeRO, xusuusta gobolka Adam, isu imaatinka halbeegyada, dhimista tartiib tartiib ah, xulashooyinka rarista, iyo kharashyada isgaarsiinta.

Wadooyinka haga ee la xidhiidhaEber Iyo Hagaajinta Sharded
Su'aal 1 ee 8

Waa maxay dib-u-celinta ZeRO marka la barbar dhigo isbarbardhigga xogta cad?

Horudhac su'aal kasta8 su'aalo

Jawaabaha iyo sharraxaadaha waxay soo baxayaan markaad qaadanayso kediska.

  1. Waa maxay dib-u-celinta ZeRO marka la barbar dhigo isbarbardhigga xogta cad?
    • Ku koobbi koobiyada buuxa ee gobolka wanaajinta, gradients, iyo cabbirada GPU kasta
    • Nuqullada xogta tababarka ee GPU kasta
    • Xadhkaha shabakada oo aan badnayn oo u dhexeeya qanjidhada
    • Hordhac soo noqnoqda ayaa dulmaraya isla dufcaddii
  2. Waa maxay sababta ay optimizer state inta badan u tahay hog xusuusta ugu weyn ee Adam?
    • Aadan ma kaydiyo xaalad dheeraad ah haba yaraatee
    • Aadam waxa uu hayaa qiimayaal dheeraad ah oo halbeeg kasta ah (sida dardargelinta iyo kala duwanaanshiyaha), badiyaa dhawr jeer cabbirka qaabka
    • Aadam waxa uu nuqul ka dhigayaa xogta oo dhan
    • Aadam waxa uu u baahan yahay in uu kaydiyo wax kasta oo soo dhaafay weligiis
  3. Waa maxay jeexjeexyada ZeRO Stage 3 ee heerarka 1 iyo 2 aysan samaynin?
    • Xogta tababarka
    • Shaqada luminta
    • Qaabka ayaa cabbiraya laftooda
    • Jadwalka heerka waxbarashada
  4. Marxaladda ZeRO 3, sidee buu GPU-gu u helayaa cabbirrada buuxa ee uu uga baahan yahay gudbinta lakabka?
    • Marna uma baahna cabbirada buuxa
    • Waxay ka soo dejisaa internetka
    • Waxay dib uga xisaabisaa meel eber ah mar kasta
    • Waxay qabataa isku-ururin si ay u ururiso jeexjeexyada maqan, ka dibna tuurto iyaga ka dib
  5. Waa kuwee astaanta PyTorch ee asal ahaan u fulisa shaxaynta qaabka ZeRO?
    • FSDP (Isbarbardhigga Xogta La Wadaageeyey)
    • DataLoader
    • TorchScript
    • Autograd
  6. Waa maxay ganacsiga xudunta u ah ZeRO uu sameeyo si uu u badbaadiyo xusuusta?
    • Waxay u huraysaa saxnaanta tusaalaha xawaaraha
    • Waxay ku darsataa isgaarsiin dheeraad ah si ay u ururiso oo u kala firdhiso jeexjeexyada haddii loo baahdo
    • Waxay ka saartaa lakabyada qaabka
    • Waxa kaliya oo la shaqeeya moodooyinka aadka u yaryar
  7. Which state is partitioned in ZeRO Stage 1?
    • Only parameters, with optimizer states replicated
    • Parameters, gradients, and optimizer states all sharded
    • Optimizer states, while parameters and gradients remain replicated
    • Only input activations and labels
  8. What is a practical cost of ZeRO-style sharding?
    • It removes the need to calculate gradients
    • It makes every optimizer update local with no data exchange
    • It requires the model to be retrained from scratch for every step
    • Additional communication to gather parameters and distribute gradients