Lookhead and Lion Optimizers
Lookahead da Lion su ne juyi na zamani guda biyu akan inganta hanyar sadarwa.
Dubawa
Lookahead wraps any base optimizer with 'slow' and 'fast' weights for more stable progress, while Lion (EvoLved Sign Momentum) was discovered by an AI program search and updates weights using only the sign of a momentum term — making it memory-light and often faster than Adam.
Zurfafa nutsewa
Lookahead, wanda Zhang, Hinton da abokan aiki suka gabatar a shekarar 2019, yana gudanar da ma'auni na 'sauri' mai ingantawa (kamar Adam ko SGD) don matakan k, sannan ya nudge wani nau'in ma'aunin nauyi na 'hankali' kadan daga hanyar zuwa inda ma'aunin sauri ya ƙare. Wannan yana damun oscillations kuma yana rage hankali ga hyperparameters. Zaki, wanda Google ya buga a cikin 2023, ya fito daga bincike na alama akan abubuwan ingantawa. Yana bin ƙwaƙƙwara amma yana aiwatar da aikin alamar zuwa sabuntawa, don haka kowane ma'auni yana motsawa ta ƙayyadadden girman mataki zuwa ga tarin alamar gradient. Zaki yana adana madaidaicin buffer kawai (rabin jihar Adamu, wanda ke riƙe biyu), yana amfani da lalata nauyi mafi girma da ƙarancin koyo, kuma ya daidaita ko ya doke Adamu akan manyan hangen nesa da ƙirar harshe yayin horo cikin sauri da rahusa.
Fahimtar Fasaha
Duba kai: bayan k matakai masu sauri suna samar da ma'auni θ_sauri, jinkirin ma'aunin nauyi yana tafiya kamar φ ← φ + α(θ_fast - φ), sannan mai saurin ingantawa yana sake saitawa zuwa φ. Sabuntawar zaki: m ← β1·m + (1-β1) · g don interpolation, amma matakin nauyi shine θ ← θ - η · (alama (β2 · m + (1-β2) · g) + λθ). Aikin alamar yana yin yumfom ɗin ɗaukaka sabuntawar kowane haɗin gwiwa, wanda ke aiki kamar daidaitacce kuma yana bayyana dalilin da yasa zaki yana buƙatar ƙaramin koyo fiye da Adamu.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Lookhead and Lion Optimizers
An karɓi Lion a cikin manyan ayyukan horarwa da yawa saboda yana yanke ƙwaƙwalwar ingantawa kuma yana iya hanzarta haɗuwa, kuma bincikensa yana nuna binciken algorithm mai sarrafa kansa na 'AI-designing-AI' azaman ainihin tushen fa'ida. Yi tsammanin ƙarin abubuwan ingantawa waɗanda aka samo asali, tsare-tsare masu haɗaka waɗanda ke haɗa nau'ikan jinkirin salon Lookhead tare da sabunta tushen alamar, da haɓaka sha'awar masu inganta ƙwaƙwalwar ajiya kamar yadda girman ƙirar ke ci gaba da jaddada kasafin ƙwaƙwalwar GPU.
Aiwatar da Gaskiyar Duniya
Rufe Adam tare da Lookhead don daidaita horar da masu canza wuta da rage kokarin daidaita hyperparameter.
Amfani da Lion don horar da manyan nau'ikan hangen nesa (misali, ViT) tare da ƙananan ƙwaƙwalwar ingantawa fiye da Adam.
Horar da ƙirar harshe tare da Lion don cimma daidaito mai kama da ƙarancin ƙididdige farashi.
Haɗa Lookahead tare da SGD a cikin wakilai masu ƙarfafawa-koyo don sassauƙar sabunta manufofin hayaniya.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Lookahead and Lion Optimizers quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Zazzagewar Jihar Optimizer zuwa CPU da NVMe
Tambayoyin da ake yawan yi
What is Lookahead and Lion Optimizers?
Lookahead da Lion su ne juyi na zamani guda biyu akan inganta hanyar sadarwa. Lookhead yana kunshe duk wani mai inganta tushe tare da 'hankali' da 'sauri' ma'aunin nauyi don ƙarin kwanciyar hankali, yayin da Lion (EvoLved Sign Momentum) ya gano ta hanyar binciken shirin AI kuma yana sabunta ma'auni ta amfani da alamar lokaci kawai - yana mai da shi ƙwaƙwalwar ajiya-haske kuma sau da yawa sauri fiye da Adam.
Menene ma'anar ƙididdigewa mataki a cikin sabuntawar nauyi na Lion ingantawa?
Zaki yana aiwatar da aikin alamar zuwa lokaci mai tsaka-tsaki, don haka kowane siga yana motsawa ta mataki iri ɗaya a cikin alamar gradient.
Ta yaya Lookhead yake tsara ingantawarsa?
Lookhead yana gudanar da ingantaccen na'urar ciki mai sauri don matakan k, sannan yana motsa jinkirin nauyi kaɗan na hanyar zuwa ma'aunin sauri da sake saiti.
Ta yaya aka samo asali na Zakin ingantawa?
Zaki (EvoLved Sign Momentum) ya fito daga tsarin binciken shirye-shirye wanda ya samo asali da kimanta shirye-shiryen inganta ɗan takara.
Idan aka kwatanta da Adamu, menene babban fa'idar ƙwaƙwalwar ajiya na Zaki?
Adam yana bin lokacin farko da na biyu; Zaki yana riƙe da buffer guda ɗaya kawai, yana karkata kusan ƙwaƙwalwar jihar ingantawa.
Me yasa Zaki yawanci yana buƙatar ƙarancin koyo fiye da Adamu?
Tun da aikin alamar yana daidaita girman ɗaukakar kowane haɗin gwiwa, ingantaccen matakin yana da girma, don haka ana amfani da ƙaramin ƙimar koyo (da girman lalata nauyi).