Ikibazo gihuza ibibazo · Hagati Urwego

Ikizamini cya Gradient

Gerageza micro-batch gradient sums, ingano yingirakamaro yicyiciro, gupima igihombo, guhitamo igihe, imipaka isanzwe, hamwe namakosa yo gusubiramo imburagihe.

Inzira zijyanye no kuyoboraKwiyongera
Ikibazo 1 Bya 8

Niki kwirundanya buhoro buhoro bikwemerera gukora?

Reba buri kibazo8 ibibazo

Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.

  1. Niki kwirundanya buhoro buhoro bikwemerera gukora?
    • Gari ya moshi idafite GPU
    • Simbuka inyuma
    • Gereranya ingano nini yingirakamaro mugice cyo kwibuka
    • Kuraho optimizer rwose
  2. Hamwe na micro-batch ingano ya 4 na 8 yo gukusanya, ingano yicyiciro ni:
    • 32
    • 4
    • 8
    • 12
  3. With an accumulation factor N, when should optimizer.step() and zero_grad() occur?
    • Nyuma ya buri micro-batch
    • Nta na rimwe
    • Mbere yo kunyura imbere
    • Call optimizer.step() after N micro-batches, then clear gradients before the next accumulation window
  4. Kugirango ugumane igipimo cyiza cyo kwiga gikwiye, igihombo kuri buri micro-batch ni:
    • Yagwijwe na N.
    • Igabanijwe numubare wintambwe N.
    • Shyira kuri zeru
    • Byasimbuwe nukuri
  5. Ni ubuhe buryo busanzwe busanzwe bubiri hamwe no gukusanya buhoro buhoro?
    • Ibice cyangwa itsinda risanzwe, kubera ko bidashingiye kumibare yimibare
    • Batch ibisanzwe, kuko ikunda uduce duto
    • Nta bisanzwe bisanzwe bikorana nayo
    • Gusa abataye ishuri
  6. Igiciro nyamukuru cyo gukoresha buhoro buhoro ni:
    • Ububiko bwinshi bwa GPU bukoreshwa kuri peak
    • Gutakaza gradients zose
    • Igihe kirekire kurukuta-isaha mugihe cyo kuvugurura neza
    • Kudashobora gukoresha optimizer
  7. A training loop clears gradients after every micro-batch but steps the optimizer every fourth batch. What is lost?
    • The model weights are deleted
    • The earlier micro-batch contributions to the accumulated update
    • The effective batch becomes four times larger
    • The loss automatically becomes zero
  8. Why may four micro-batches with BatchNorm behave differently from one physical batch of the same total size?
    • The optimizer cannot accept four backward passes
    • Gradient addition is forbidden in BatchNorm
    • The number of training examples changes
    • BatchNorm uses statistics from each micro-batch rather than the combined batch