Lookhead na Lion Optimizers
Lookahead na ọdụm bụ ntụgharị ọgbara ọhụrụ abụọ na njikarịcha netwọkụ neural.
Nchịkọta
Lookahead wraps any base optimizer with 'slow' and 'fast' weights for more stable progress, while Lion (EvoLved Sign Momentum) was discovered by an AI program search and updates weights using only the sign of a momentum term — making it memory-light and often faster than Adam.
Ime miri emi
Lookahead, nke Zhang, Hinton na ndị ọrụ ibe ha tụrụ aro ya na 2019, na-arụ ọkọlọtọ 'ngwa ngwa' optimizer (dị ka Adam ma ọ bụ SGD) maka k nzọụkwụ, wee kpatụ ụdị dị iche iche nke 'nwayọ' arọ nke dị nta nke ụzọ ebe ibu ngwa ngwa kwụsịrị. Nke a na-ebelata oscillations ma na-ebelata mmetụta nke hyperparameters. Ọdụm, nke Google bipụtara na 2023, siri na nchọ mmemme ihe atụ n'elu algọridim kacha mma. Ọ na-akwado ngwa ngwa mana ọ na-etinye ọrụ akara na mmelite ahụ, yabụ oke ọ bụla na-aga site na nha nrịgo edobere n'akụkụ akara gradient anakọtara. Ọdụm na-echekwa naanị ihe nchekwa ọkụ ọkụ (ọkara steeti Adam, nke na-edobe abụọ), na-eji ire ere buru ibu na obere mmụta mmụta, ma dakọtara ma ọ bụ tie Adam na nnukwu ọhụụ na ụdị asụsụ mgbe ọ na-azụ ngwa ngwa na ọnụ ala.
Nghọta nka nka
Nleba anya n'isi: ka emechara usoro ngwa ngwa na-emepụta ihe ọ̀tụ̀tụ̀ θ_fast, ịdị nwayọọ na-ebu ibu na-aga ka φ ← φ + α(θ_fast - φ), mgbe ahụ ngwa ngwa optimizer na-amaliteghachi na φ. Mmelite ọdụm: m ← β1·m + (1-β1) · g maka interpolation, mana nzọụkwụ dị arọ bụ θ ← θ - η· (akara (β2·m + (1-β2)·g) + λθ). Arụmọrụ akara na-eme ka nhazi ọkwa ọ bụla bụrụ otu, nke na-eme dị ka nhazi nkịtị ma kọwaa ihe kpatara ọdụm ji chọọ ọnụ ọgụgụ mmụta dị obere karịa Adam.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke Lookhead na ndị na-ebuli ọdụm
A nakweere ọdụm n'ọtụtụ ọzụzụ ọzụzụ dị ukwuu n'ihi na ọ na-ebelata ebe nchekwa kachasị mma ma nwee ike ime ka njikọta dị ngwa ngwa, na nchọpụta ya gosipụtara ọchụchọ 'AI-designing-AI' akpaghị aka dị ka ezigbo isi iyi nke uru bara uru. Na-atụ anya ihe nchọta ndị ọzọ enwetara, atụmatụ ngwakọ nke na-agwakọta ụdị uka-isi ngwa ngwa na mmelite dabere na akara, yana mmasị na-eto eto na ndị na-ebuli ebe nchekwa na-arụ ọrụ nke ọma ka nha ụdị na-emesi mmefu ego ebe nchekwa GPU ike.
Mmejuputa n'ezie n'ụwa
Kechie Adam na Lookahead iji mee ka ọzụzụ nke ndị na-agbanwe agbanwe kwụsie ike ma belata mbọ ngbanwe hyperparameter.
Iji ọdụm na-azụ nnukwu ụdị ọhụụ (dịka ọmụmaatụ, ViT) na ebe nchekwa kachasị dị ala karịa Adam.
Iji ọdụm na-azụ ụdị asụsụ iji nweta izi ezi atụnyere na ọnụ ahịa mbelata.
Na-ejikọta Lookahead na SGD na ndị ọrụ nkwado-mmụta maka mmelite amụma mkpọtụ dị nro.
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Lookahead and Lion Optimizers quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Mbudata steeti Optimizer na CPU na NVMe
Ajụjụ a na-ajụkarị
What is Lookahead and Lion Optimizers?
Lookahead na ọdụm bụ ntụgharị ọgbara ọhụrụ abụọ na njikarịcha netwọkụ neural. Lookahead na-eji 'nwayọ' na 'ngwa ngwa' ihe ọ bụla na-eme ka ọ dị mma maka ọganihu kwụsiri ike, ebe ihe mmemme AI achọpụtara ọdụm (EvoLved Sign Momentum) ma na-emelite ihe dị arọ site na iji naanị akara nke okwu ọkụ ọkụ - na-eme ka ọ bụrụ ebe nchekwa-ìhè na mgbe ngwa ngwa karịa Adam.
Kedu ihe na-akọwa usoro mgbako na nkwalite ibu nke ọdụm?
Ọdụm na-etinye ọrụ akara ahụ n'ime nkeji oge etinyere ọnụ, yabụ oke ọ bụla na-aga site n'otu nzọ ụkwụ n'otu n'otu n'akụkụ akara gradient.
Kedu ka Lookhead si ahazi njikarịcha ya?
Lookhead na-agba ọsọ ngwa ngwa n'ime njikarịcha maka k nzọụkwụ, wee na-ebuga nwayọọ arọ obere akụkụ nke ụzọ n'ebe ndị na-ebu ngwa ngwa na-emegharị.
Kedu ka esi achọpụta na mbụ ọdụm optimizer?
Ọdụm (EvoLved Sign Momentum) siri na usoro nchọ-mmepụta welitere ma tụlere mmemme kacha mma n'ọkwa.
E jiri ya tụnyere Adam, gịnị bụ isi uru ọdụm na-echeta?
Adam na-egwu ma oge mbụ na nke abụọ; Ọdụm na-edobe naanị otu ihe nchekwa ọkụ, na-ewetu ebe nchekwa steeti kacha mma ọkara.
Gịnị kpatara ọdụm ji chọkarị obere mmụta karịa Adam?
Ebe ọ bụ na ọrụ akara na-edozi ịdị ukwuu nke mmelite nhazi ọ bụla, nzọụkwụ dị irè dị ukwuu, ya mere a na-eji obere mmụta mmụta dị ntakịrị (na nnukwu ire ere).