Stochastic gradient mgbada nwere oge
Oge bụ tweak na mgbada gradient nke na-achịkọta ọnụ ọgụgụ na-agba ọsọ nke gradients gara aga, na-ahapụ njikarịcha na-aga ngwa ngwa site na ndagwurugwu wee mee ka ọ daa jụụ.
Nchịkọta
It is one of the most widely used training tricks in deep learning.
Ime miri emi
Plain stochastic gradient descent (SGD) na-emelite paramita site n'itinye aka na ntụzịaka nke chere obere gradient dị ugbu a. N'okirikiri okirikiri dị ka ogologo ndagwurugwu dị warara, zigzag a gafere mgbidi ndị dị nrịgo ka ọ na-akpụ akpụ n'akụkụ ala dị nro. Oge, nke Polyak na-ewu ewu na nke Rumelhart na ndị ọrụ ibe ya mechara, na-edozi nke a site n'ịkwado vektọ ọsọ: nzọụkwụ ọ bụla na-agwakọta gradient ọhụrụ na obere akụkụ (ọnụọgụ ọkụ ọkụ, mgbe mgbe 0.9) nke ọsọ gara aga. Ntuziaka gradient na-agbanwe agbanwe na-ewusi ma mee ngwa ngwa, ebe ihe ndị na-emegharị emegharị na-akagbu akụkụ ụfọdụ. Ihe atụ nke anụ ahụ bụ bọọlụ dị arọ na-awụda n'ala: ọ na-ewuli ọsọ na ntụzịaka kwụ ọtọ ma ọ na-adịchaghị emegharị ya site na mkpọtụ mkpọtụ, na-enye ngwa ngwa, nhịahụ dị nro karịa vanilla SGD.
Nghọta nka nka
Mmelite ahụ na-edobe ọsọ v nke emelitere dị ka v = beta * v + gradient, mgbe ahụ, parameters na-aga site na mwepu oge ọnụego mmụta v. Site na ọnụọgụ ọnụọgụ beta, nzọụkwụ dị irè na ntụziaka na-agbanwe agbanwe na-abawanye n'ụzọ siri ike site na 1/(1 - beta); na beta = 0.9 nke dị ihe dị ka ugboro iri. Nke a na mgbakọ na mwepụ bụ nkezi gradients na-akpụ akpụ, na-eme ka mkpọtụ obere ogbe na-eme ka ọ na-echekwa ntụzịaka mgbada.
Mmetụta atụmatụ
Mkpebi doro anya
Ọ na-enyere gị aka ikewapụta nkwupụta ọrụ aka doro anya na asụsụ ahịa.
Ọnụ ego na mmefu ego
Ị nwere ike ịjụ ajụjụ mmejuputa iwu ka mma tupu itinye ego ma ọ bụ oge.
Team na usoro ọrụ
Ndị otu nwere nghọta na-eme ka ngwaahịa, amụma na mkpebi mmụta ka mma.
Ọdịnihu nke mgbada Stochastic Gradient nwere oge
Oge na-anọgide na ntọala: ndị na-eme mgbanwe dị ka Adam na ụdị ya dị iche iche na-etinye atụmatụ nke oge mbụ, na SGD nwere ume ka bụ ntọala siri ike nke na-emekarị ka ọ dị mma karịa ụzọ mgbanwe na ụdị ọhụụ buru ibu. Nchọcha na-aga n'ihu na nhazi oge n'ike, emebi emebi ibu arọ, yana mmekọrịta ya na ọzụzụ ogbe buru ibu. Na-atụ anya ka ọkụ ga-abụ isi akụrụngwa ka ndị na-ebuli elu na-etolite maka ụdị ndị buru ibu mgbe niile.
Mmejuputa n'ezie n'ụwa
Ọzụzụ netwọkụ mmekọrịta miri emi dị ka ResNet, ebe SGD nwere ume 0.9 bụ usoro nhazi ọkọlọtọ.
Atụmatụ gradient na-eme ka ọ dị nro mgbe ị na-eji obere obere batches.
Ịgbanarị ala dị larịị na-emighị emi site n'ibute ọsọ ọsọ na mpaghara dị larịị.
Na-eje ozi dị ka okwu ọkụ ọkụ n'ime ndị na-eme mgbanwe dị ka Adam na RMSprop variants.
Ihe ize ndụ & okporo ụzọ nche
Otu dị iche iche nwere ike iji otu okwu ahụ mee ihe n'ụzọ dị iche, yabụ kọwapụta oge n'oge.
Ihe nrịbama nwere ike ịdị ike ebe arụmọrụ ụwa na-adaghị adaba.
Ileghara ogo data na atụmatụ nyocha anya na-emepụtakarị nsonaazụ na-adịghị mma.
Map mmejuputa
Malite na nkọwa asụsụ dị larịị nke nsonaazụ ịchọrọ.
Họrọ otu metrik ịga nke ọma na otu ọnọdụ ọdịda tupu nnwale.
Gbaa obere onye na-anya ụgbọ elu nwere data nnọchite anya, ọ bụghị ihe ngosi ngosi na-egbu maramara.
Detuo ebe Stochastic Gradient Descent na Momentum na-enyere aka yana ebe ụzọ ndị dị mfe ka mma.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Stochastic Gradient Descent with Momentum quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Ọdịda gradient
Ajụjụ a na-ajụkarị
What is Stochastic Gradient Descent with Momentum?
Oge bụ tweak na mgbada gradient nke na-achịkọta ọnụ ọgụgụ na-agba ọsọ nke gradients gara aga, na-ahapụ njikarịcha na-aga ngwa ngwa site na ndagwurugwu wee mee ka ọ daa jụụ. Ọ bụ otu n'ime aghụghọ ọzụzụ a na-ejikarị eme ihe na mmụta miri emi.
Kedu ihe okwu mkpali na-agbakọ n'oge ọzụzụ?
Oge a na-ejigide vector ọsọ bụ nkezi na-akpụ akpụ nke gradients na-adịbeghị anya, na-eme ka ntụzịaka mmelite dị nro.
N'ime ndagwurugwu dị ogologo, dị warara, kedu nsogbu SGD dị larịị na-ata ahụhụ nke ahụ na-enyere aka idozi?
Na-enweghị ike, SGD na-efegharị n'akụkụ ụzọ mgbada nke ndagwurugwu. Oge na-akagbu oscillations ndị a ma na-agba ọsọ n'akụkụ ala ndagwurugwu dị nro.
Kedu ntụnyere anụ ahụ ka a na-ejikarị akọwa ọkụ ọkụ?
A na-atụnyere oge dị ka bọọlụ dị arọ nke na-ewuli ọsọ na ntụzịaka na-agbanwe agbanwe ma na-egbochi mgbapụ site na mkpọtụ mkpọtụ.
Ihe dị ka ole ka ọkụ ọkụ na-eme ka nzọụkwụ dị irè na ntụzịaka na-agbanwe agbanwe mgbe beta = 0.9?
Ihe nrịbawanye elu bụ ihe dịka 1/(1 - beta), na 1/(1 - 0.9) = 10, ya mere ntụzịaka na-agbanwe agbanwe na-agbago ihe ruru okpukpu iri.
Kedu ihe njikarịcha ọgbara ọhụrụ na-etinye atụmatụ nlebara anya nke oge mbụ?
Adam na-ejikọta ọnụ ọgụgụ dị ka nkeji oge mbụ (nke pụtara) nke gradients na atụmatụ nkeji nke abụọ (iche) maka nhazi mgbanwe.