Ubuyobozi bwa tekiniki

Kwiyongera

Kwiyegeranya kwa Gradient bigufasha kwigana ingano nini yububiko bwa GPU bugarukira mugukusanya gradients hejuru ya mini-mato mato mbere yo kuvugurura uburemere.

Umva iki gitaboAI-byavuzwe · ~2 min

Kuriyi page2 min soma
  1. Incamake
  2. Kwibira cyane
  3. Gushyira mu bikorwa Isi
  4. Igihe kizaza cyo Kwiyongera
  5. Kubishyira mubikorwa
  6. Komeza Ubushakashatsi
  7. Ibibazo bikunze kubazwa

Incamake

Nibikorwa bisanzwe byo guhugura moderi nini mugihe kwibuka ari icyuho.

Kwibira cyane

Mubisanzwe intambwe yimyitozo itunganya icyiciro kimwe, ikabara gradients, igahita ivugurura ibipimo. Hamwe no kwegeranya gahoro gahoro, ukoresha inzira nyinshi imbere ninyuma yinyuma kuri micye-mato mato, ukongeraho gradients zabo hamwe muri parameter ya buffer, hanyuma ugahamagara gusa intambwe ya optimizer (na zeru gradients) nyuma ya N micro-batch. Ingano yingirakamaro yicyiciro ihinduka micro-batch inshuro N, nubwo ububiko bwo hejuru bwonyine bufite micro-batch yibikorwa. Ibi bifite akamaro kuko ibisubizo byinshi byamahugurwa bifata ibyiciro byinshi kumibare ihamye, kandi kubera ko moderi nka transformateur nini idashobora guhuza intego yuzuye kubikoresho bimwe. Gufata: icyiciro-gisanzwe imibare ibarwa kuri micro-batch, bityo amahame ya layer cyangwa itsinda risanzwe ryiza hamwe no kwegeranya, kandi ugomba gupima igihombo neza kugirango ugumane igipimo cyiza cyo kwiga neza.

Gushyira mu bikorwa Isi

Kuringaniza neza urugero rwururimi runini kumuguzi umwe GPU ukusanya mikoro irenga 8 cyangwa 16 kugirango ugere ku gice cyiza cya magana.

Guhugura ibyerekezo bihanitse cyane cyangwa ibyiciro byerekana aho ndetse nicyiciro cya 2 gihuye, ariko resept ikenera icyiciro cyiza cya 32.

Guhobera Isura Yumutoza hamwe na PyTorch Umurabyo werekana igenamigambi_gukurikirana_ibikorwa byakoreshejwe muburyo busanzwe-VRAM.

Kubyara impapuro nini-nini ibisubizo kubikoresho bito muguhuza ingano yicyiciro cyiza binyuze mukwirundanya.

Igihe kizaza cyo Kwiyongera

Kwiyongera kwa Gradient bizagumaho nkibisanzwe nkubunini bwikitegererezo burenze igikoresho kimwe cyibikoresho. Igenda ihuza byinshi bivanze neza, kugenzura ibikorwa, kugenzura ZeRO, hamwe no guhuza imiyoboro muburyo nka DeepSpeed ​​na FSDP. Witegereze kwihuta cyane aho amasomero yikora-yo guteranya intambwe yo gukusanya ingengo yimikorere yo kwibuka, kandi ugakomeza akamaro ko guhuza neza moderi nini ku byuma byoroheje, harimo na GPU y'abaguzi aho ifungura amahugurwa bitashoboka.

Kubishyira mubikorwa

Impamvu ari ngombwa

  • Igiciro na bije. Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
  • Ibyemezo bisobanutse. Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
  • Kugenzura ubuziranenge. Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.

Witondere

  • Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
  • Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
  • Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.

Intambwe ikurikira

  1. Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
  2. Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
  3. Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
  4. Tegura inzira yo gusubiza ibyabaye mbere yo gupima.

Komeza Ubushakashatsi

Ibaruwa y'amakuru y'ubuntu

Ibuka amakuru ya buri munsi ya AI

Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.

Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.

Isuzume

Take the Gradient Accumulation quiz

Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.

Tangira ikibazo

Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano

Ibibazo bikunze kubazwa

Kwiyongera kwa Gradient ni iki?

Kwiyegeranya kwa Gradient bigufasha kwigana ingano nini yububiko bwa GPU bugarukira mugukusanya gradients hejuru ya mini-mato mato mbere yo kuvugurura uburemere. Nibikorwa bisanzwe byo guhugura moderi nini mugihe kwibuka ari icyuho.

Niki kwirundanya buhoro buhoro bikwemerera gukora?

Mugukusanya gradients hejuru ya micro-batch mbere yo kuvugurura, wigana igice kinini utagifashe byose murwibutso icyarimwe.

With an accumulation factor N, when should optimizer.step() and zero_grad() occur?

Accumulate gradients across N micro-batches, call optimizer.step() once for the effective batch, then clear gradients before the next window; begin training with cleared buffers.

Ni ubuhe buryo busanzwe busanzwe bubiri hamwe no gukusanya buhoro buhoro?

Batch-norm ibara imibare kuri micro-batch, so layer cyangwa group group irinda kudahuza na micro-batch.

Why may four micro-batches with BatchNorm behave differently from one physical batch of the same total size?

Batch-dependent forward computations break exact equivalence.