Ikibazo gihuza ibibazo · Hagati Urwego
Ikizamini cya Gradient
Gerageza micro-batch gradient sums, ingano yingirakamaro yicyiciro, gupima igihombo, guhitamo igihe, imipaka isanzwe, hamwe namakosa yo gusubiramo imburagihe.
Ikibazo 1 Bya 8
Niki kwirundanya buhoro buhoro bikwemerera gukora?
Reba buri kibazo8 ibibazo
Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.
- Niki kwirundanya buhoro buhoro bikwemerera gukora?
- Gari ya moshi idafite GPU
- Simbuka inyuma
- Gereranya ingano nini yingirakamaro mugice cyo kwibuka
- Kuraho optimizer rwose
- Hamwe na micro-batch ingano ya 4 na 8 yo gukusanya, ingano yicyiciro ni:
- 32
- 4
- 8
- 12
- With an accumulation factor N, when should optimizer.step() and zero_grad() occur?
- Nyuma ya buri micro-batch
- Nta na rimwe
- Mbere yo kunyura imbere
- Call optimizer.step() after N micro-batches, then clear gradients before the next accumulation window
- Kugirango ugumane igipimo cyiza cyo kwiga gikwiye, igihombo kuri buri micro-batch ni:
- Yagwijwe na N.
- Igabanijwe numubare wintambwe N.
- Shyira kuri zeru
- Byasimbuwe nukuri
- Ni ubuhe buryo busanzwe busanzwe bubiri hamwe no gukusanya buhoro buhoro?
- Ibice cyangwa itsinda risanzwe, kubera ko bidashingiye kumibare yimibare
- Batch ibisanzwe, kuko ikunda uduce duto
- Nta bisanzwe bisanzwe bikorana nayo
- Gusa abataye ishuri
- Igiciro nyamukuru cyo gukoresha buhoro buhoro ni:
- Ububiko bwinshi bwa GPU bukoreshwa kuri peak
- Gutakaza gradients zose
- Igihe kirekire kurukuta-isaha mugihe cyo kuvugurura neza
- Kudashobora gukoresha optimizer
- A training loop clears gradients after every micro-batch but steps the optimizer every fourth batch. What is lost?
- The model weights are deleted
- The earlier micro-batch contributions to the accumulated update
- The effective batch becomes four times larger
- The loss automatically becomes zero
- Why may four micro-batches with BatchNorm behave differently from one physical batch of the same total size?
- The optimizer cannot accept four backward passes
- Gradient addition is forbidden in BatchNorm
- The number of training examples changes
- BatchNorm uses statistics from each micro-batch rather than the combined batch