Ikibazo gihuza ibibazo · Biragoye Urwego

Ikibazo cya ZeRO na Sharded Optimizers Ikibazo

Gupfukirana ibyiciro bya ZeRO, kwibuka leta ya Adam, gukusanya ibipimo, kugabanya gahoro gahoro, guhitamo ibicuruzwa, hamwe nigiciro cyitumanaho.

Inzira zijyanye no kuyoboraZeru na Sharded Optimizers
Ikibazo 1 Bya 8

Ni ubuhe busumbane ZeRO ikuraho ugereranije namakuru asobanutse?

Reba buri kibazo8 ibibazo

Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.

  1. Ni ubuhe busumbane ZeRO ikuraho ugereranije namakuru asobanutse?
    • Wigane kopi yuzuye ya optimizer leta, gradients, nibipimo kuri buri GPU
    • Gukoporora kopi yamakuru yamahugurwa kuri buri GPU
    • Imiyoboro y'urusobe rwinshi hagati yumutwe
    • Gusubiramo imbere byanyuze hejuru yicyiciro kimwe
  2. Ni ukubera iki optimizer leta ikunda kwibuka cyane hamwe na Adam?
    • Adam ntabika leta yinyongera na gato
    • Adam agumana inyongera kuri buri kintu (nkumuvuduko no gutandukana), inshuro nyinshi ubunini bwikitegererezo
    • Adam yigana imibare yose
    • Adam arasaba kubika buri cyiciro cyahise
  3. Niki ZeRO Icyiciro cya 3 shard Icyiciro cya 1 nicya 2 bidakora?
    • Amakuru y'amahugurwa
    • Igikorwa cyo gutakaza
    • Icyitegererezo cy'icyitegererezo ubwacyo
    • Gahunda yo kwiga
  4. Muri ZeRO Icyiciro cya 3, nigute GPU ibona ibipimo byuzuye ikenera kugirango igere imbere?
    • Ntabwo ikenera ibipimo byuzuye
    • Irabakura kuri enterineti
    • Irabisubiramo guhera buri gihe
    • Irakora byose-ikusanya gukusanya ibice byabuze, hanyuma ikajugunya nyuma
  5. Nibihe PyTorch biranga kavukire bishyira mubikorwa bya ZeRO?
    • FSDP (Ikigereranyo Cyuzuye Cyuzuye)
    • DataLoader
    • TorchScript
    • Autograd
  6. Nubuhe bucuruzi bwibanze ZeRO ikora kugirango ubike ububiko?
    • Itanga icyitegererezo cyukuri kubwihuta
    • Yongera itumanaho ryiyongera kugirango akusanyirize hamwe kandi akwirakwize ibice nkuko bikenewe
    • Ikuraho ibice kuri moderi
    • Gusa ikorana na moderi nto cyane
  7. Which state is partitioned in ZeRO Stage 1?
    • Only parameters, with optimizer states replicated
    • Parameters, gradients, and optimizer states all sharded
    • Optimizer states, while parameters and gradients remain replicated
    • Only input activations and labels
  8. What is a practical cost of ZeRO-style sharding?
    • It removes the need to calculate gradients
    • It makes every optimizer update local with no data exchange
    • It requires the model to be retrained from scratch for every step
    • Additional communication to gather parameters and distribute gradients