Kwiyongera
Kwiyegeranya kwa Gradient bigufasha kwigana ingano nini yububiko bwa GPU bugarukira mugukusanya gradients hejuru ya mini-mato mato mbere yo kuvugurura uburemere.
Incamake
It is the standard workaround for training big models when memory is the bottleneck.
Kwibira cyane
Mubisanzwe intambwe yimyitozo itunganya icyiciro kimwe, ikabara gradients, igahita ivugurura ibipimo. Hamwe no kwegeranya gahoro gahoro, ukoresha inzira nyinshi imbere ninyuma yinyuma kuri micye-mato mato, ukongeraho gradients zabo hamwe muri parameter ya buffer, hanyuma ugahamagara gusa intambwe ya optimizer (na zeru gradients) nyuma ya N micro-batch. Ingano yingirakamaro yicyiciro ihinduka micro-batch inshuro N, nubwo ububiko bwo hejuru bwonyine bufite micro-batch yibikorwa. Ibi bifite akamaro kuko ibisubizo byinshi byamahugurwa bifata ibyiciro byinshi kumibare ihamye, kandi kubera ko moderi nka transformateur nini idashobora guhuza intego yuzuye kubikoresho bimwe. Gufata: icyiciro-gisanzwe imibare ibarwa kuri micro-batch, bityo amahame ya layer cyangwa itsinda risanzwe ryiza hamwe no kwegeranya, kandi ugomba gupima igihombo neza kugirango ugumane igipimo cyiza cyo kwiga neza.
Ubushishozi
Kuberako gradients yigihombo cyegeranijwe ari inyongera, kwegeranya gradients hejuru ya N micro-batches ni imibare ihwanye nicyiciro kinini, uramutse ugereranije neza. Gushyira mubikorwa mubisanzwe bigabanya buri gihombo cya micro-bat na N mbere yo gusubira inyuma, bityo gradient yegeranijwe ihwanye nuburyo hejuru yicyiciro cyiza. Urasimbuka optimizer.step () na zeru_grad () kugeza kuri Nth micro-batch, ucuruza igihe cyo kubara igihe cyo kugabanya ububiko bwibuke.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Igihe kizaza cyo Kwiyongera
Kwiyongera kwa Gradient bizagumaho nkibisanzwe nkubunini bwikitegererezo burenze igikoresho kimwe cyibikoresho. Igenda ihuza byinshi bivanze neza, kugenzura ibikorwa, kugenzura ZeRO, hamwe no guhuza imiyoboro muburyo nka DeepSpeed na FSDP. Witegereze kwihuta cyane aho amasomero yikora-yo guteranya intambwe yo gukusanya ingengo yimikorere yo kwibuka, kandi ugakomeza akamaro ko guhuza neza moderi nini ku byuma byoroheje, harimo na GPU y'abaguzi aho ifungura amahugurwa bitashoboka.
Gushyira mu bikorwa Isi
Kuringaniza neza urugero rwururimi runini kumuguzi umwe GPU ukusanya mikoro irenga 8 cyangwa 16 kugirango ugere ku gice cyiza cya magana.
Guhugura ibyerekezo bihanitse cyane cyangwa ibyiciro byerekana aho ndetse nicyiciro cya 2 gihuye, ariko resept ikenera icyiciro cyiza cya 32.
Guhobera Isura Yumutoza hamwe na PyTorch Umurabyo werekana igenamigambi_gukurikirana_ibikorwa byakoreshejwe muburyo busanzwe-VRAM.
Kubyara impapuro nini-nini ibisubizo kubikoresho bito muguhuza ingano yicyiciro cyiza binyuze mukwirundanya.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Gradient Accumulation quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Kurimbuka no Guturika Gradients
Ibibazo bikunze kubazwa
What is Gradient Accumulation?
Kwiyegeranya kwa Gradient bigufasha kwigana ingano nini yububiko bwa GPU bugarukira mugukusanya gradients hejuru ya mini-mato mato mbere yo kuvugurura uburemere. Nibikorwa bisanzwe byo guhugura moderi nini mugihe kwibuka ari icyuho.
Niki kwirundanya buhoro buhoro bikwemerera gukora?
Mugukusanya gradients hejuru ya micro-batch mbere yo kuvugurura, wigana igice kinini utagifashe byose murwibutso icyarimwe.
Mugihe cyo kwegeranya, ni ryari uhamagara intambwe ya optimizer na zeru gradients?
Ukusanya gradients kuri N micro-batch hanyuma ukavugurura rimwe gusa kurangiza ukwezi.
Ni ubuhe buryo busanzwe busanzwe bubiri hamwe no gukusanya buhoro buhoro?
Batch-norm ibara imibare kuri micro-batch, so layer cyangwa group group irinda kudahuza na micro-batch.