Lookahead ak gaynde yiy gëna xéewale
Lookahead ak Lion ñaari coppite yu bees lañu ci gëna baaxal reso neuronal yi.
Résumé
Lookahead dafay laxas bépp optimisatëru base ak diisaay yu 'yeex' ak 'gaaw' ngir gëna dëgër, ci noonu la Lion (EvoLved Sign Momentum) ñu gis ko ci prograamu IA buy seet ak yeesal diisaay yi di jëfandikoo màndarga momentum rek - muy def ko memory-light te lu bari lu gëna gaaw Adam.
Plongeur bu xóot
Lookahead, bi Zhang, Hinton ak ay naataango taxawal ci 2019, dafay doxal benn 'gaaw' optimizer (lu melni Adam wala SGD) ngir k jéego, ba noppi nudges benn set bu 'yeex' poids benn fraction ci yoon wi jëm ci fi poids gaaw yi mujjee. Loolu dafay wàññi osilaasioŋ yi ba noppi wàññi sensitiwite ci hiperparametre yi. Lion, bi Google siiwal ci 2023, mingi génne ci seetug prograamu màndarga ci kaw algorithm yiy gëna xéewale. Dafay topp momentum bi waaye dafay jëfandikoo fonction sign bi ci yeesal bi, suko defee parametre bu nekk di toxu ci dayo jéego bu takku ci wàllu sign gradient biñ dajale. Lion du denc ludul tampon momentum (genn-wàll nekkinu Adam, biy denc ñaar), jëfandikoo decay bu gëna mag ak njàng mu gëna ndaw, ba noppi match wala beat Adam ci gis-gis bu yaatu ak xeetu làkk yi ci di tàggat lu gëna gaaw te gëna yomb.
Gis-gis xarala
Coppite ci kanam: ginaaw k jéego yu gaaw yu defar ay poid θ_fast, poid yu yeex yi dañuy toxu ni φ ← φ + α(θ_fast − φ), ginaaw ga optimisatëru gaaw bi dafay reset ci φ. yeesal gaynde: m ← β1 · m + (1-β1) · g ngir interpolaasioŋ, waaye jéego diisaay bi mooy θ ← θ - η · (màndarga (β2 · m + (1-β2) · g) + λθ). Liggéeyu màndarga dafay tax bepp yeesal magnitude coordonnée nekk benn, loolu dafay nuru ab normalisasioŋ bu nëbbu te leeral lu tax Lion soxla taxawaayu jàng bu gëna ndaw Adam.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu Lookahead ak gaynde yiy gëna xéewale
Lion dañu ko jëfandikoo ci ay tàggat yu bari ndax dafay dagg memory optimisateur bi te mën na gaawal convergence bi, te gis-gis bimu am dafay wane seetlu algorithm 'AI-designing-AI' bi nekk luy am njariñ ci liggéey bi. Xaarandil yeneen optimisatër yu bawoo ci seet, ay pexe hybrid yuy jaxase poid yu yeex yu Lookahead ak yeesali siñaal, ak bëgg-bëgg buy gëna yokk ci optimisatër yu am mémoire ndax dayo model yi dañuy wéy di stress ci budget mémoire GPU.
Doxal ci àdduna dëgg
Defar Adam ak Lookahead ngir dakkal tàggat trafo yi ak wàññi coono bi ci ajustement hyperparamètre yi.
Jëfandikoo Lion ngir tàggat ay xeetu gis-gis yu rëy (lu melni, ViT) ak mémoire optimiser bu gëna ndaw Adam.
Tàggat modeli làkk yi ak Lion ngir am njubte bu méngoo ak njëgu ordinatër bu wàññeeku.
Boole Lookahead ak SGD ci ndaw yiy jàngale doole ngir mëna yeesal politik yu bari xumbaay yi.
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Lookahead and Lion Optimizers quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Optimiser bi dafay wàcci ci CPU ak NVMe
Laaj yi ñuy faral di laaj
Luy Lookahead ak gaynde yiy gëna xéewale?
Lookahead ak Lion ñaari coppite yu bees lañu ci gëna baaxal reso neuronal yi. Lookahead dafay laxas bépp optimisatëru base ak diisaay yu 'yeex' ak 'gaaw' ngir gëna dëgër, ci noonu la Lion (EvoLved Sign Momentum) ñu gis ko ci prograamu IA buy seet ak yeesal diisaay yi di jëfandikoo màndarga momentum rek - muy def ko memory-light te lu bari lu gëna gaaw ci Adam.
Lan mooy jéego bi gëna am solo ci yeesali diisaayu Lion optimizer?
Lion dafay jëfandikoo fonction siñaal bi ci ab terme momentum buñ interpole, suko defee parametre bu nekk di toxu ci jéego bu maase ci wàllu siñaalu gradient bi.
naka la Lookahead di defaree njariñam?
Lookahead dafay doxal ab optimisatëru biir bu gaaw ngir k jéego, ba noppi mu toxal poid yu yeex yi ci yoon wi jëm ci poid yu gaaw yi ba noppi reset.
naka lañu njëkka gisee Lion optimizer?
Lion (EvoLved Sign Momentum) mingi juddoo ci ab doxalinu seetlu prograam bu jëm kanam ba noppi jàngat prograam yiy gëna xéewale.
Buñu ko méngale ak Adam, lan mooy njariñu fàttaliku bu mag bi Lion am?
Adam topp na saa yu njëkk yi ak yu ñaareel yi; Lion benn tampon momentum kese lay denc, lu tollu ci genn-wàll memory stade optimisateur bi.
Lan moo waral Lion di soxla njàng mu gëna ndaw Adam?
Bi nga xamee ni fonction signe bi dafay tëral magnitude bu yeesal coordonnée bu nekk, jéego bi am solo dafa yaatu, moo tax ñuy jëfandikoo tolluwaayu jàng bu gëna ndaw (ak decay bu gëna mag).