SwiGLU ak liggéey yu am tëju
SwiGLU fonction aktivasioŋ la buy yokk benn projection ligneer bu duggal bi ak ñaareelu projection buñ aktive ci Swish, muy liggéey ni buntu buñ mëna jàng, bu aju ci done yi ci biir transformatër biy feed-forward layers.
Résumé
It consistently improves language-model quality, which is why nearly every modern LLM uses it.
Plongeur bu xóot
Benn bloc feed-forward transformateur buñ miin mooy ñaari couche ligneaire yu am ReLU wala GELU ci digg bi. Dauphin ak ñeneen ñi ñoo ko xalaat. ci 2016, xaaj projection bu njëkk bi ñaari xaaj nga jëfandikoo benn xaaj bi ngir buntu beneen bi jaaraleko ci yokk-wise élément. SwiGLU, bi Noam Shazeer siiwal ci 2020, dafay jëfandikoo fonction Swish (SiLU) ngir buntu boobu: genn = (Swish (xW) * (xV)) W2, ak ñetti matris yu poid ci barabu ñaar. Gating bi dafay may reso bi mu tànnee jàll wala dindi xibaar ci dimension bu nekk. Ndax yokk ñatteelu matrix bi dafay màgg ay parametre, jëfandikoo gi dafay wàññi dimension bu nëbbu bi ba ci ñaar ci ñett, suko defee xayma bi yépp nekk luy méngoo ak GELU MLP. Jàngat yu Shazeer dañu wane njariñu jaxasoo buñ mëna natt, LLaMA, PaLM, ak Mistral ñoom ñépp jël nañu ko.
Gis-gis xarala
Swish mooy x * sigmoid (beta * x), muy fonction bu nooy, bu amul monotone, te wuute na ak ReLU, dafay may valeur négatif yu ndaw ñu jaar. Ci SwiGLU, bànxaasu 'gate' bi Swish(xW) dafay defar ay valeur yu jege 0 wala 1 yuy yokk 'valeur' bànxaasu xV bi ci wàllu elemen, kon bépp bànxaas bu nëbbu bi dañu koy moduler ci siñaal buñ jàng, bu aju ci dugal. Ñatteelu matrix bi mooy njëg bi; ñaar ci ñett yu nëbbu yi dañuy tëye budget FLOP bi méngoo ak vanille feed-forward layer.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu SwiGLU ak Gated
SwiGLU dafa nekk MLP biñ jagleel LLM yu ubbeeku yi te daanaka duñu ko dindi ci diir bu gàtt. Yoon yi am solo ñooy GeGLU ak ReGLU, kernel GPU yuñ boole ngir xayma ñaari projection yi ci benn yoon, ak boole MLP yu am buntu ak njaxasu-ekspert suko defee eksper bu nekk ci boppam nekk benn bloku SwiGLU. Gëstukat yi ñu ngi jàngat itam li waral gating di jàppale optimisation, ngir sos gate yu gëna xéewale.
Doxal ci àdduna dëgg
LLaMA, PaLM, ak Mistral wecci GELU feed-forward couche ak SwiGLU ngir wàññi jaaxle ci xayma bu tolloo
Dimension bu nëbbu bi dañu ko yokk ba tollu ci ñaar ci ñett (8/3 d) suko defee matrix gating bi gëna yokk du gonfle FLOPs
Royuwaayi njaxasu-ekspert yu melni Mixtral jëfandikoo ay blok SwiGLU ni reso feed-forward bu eksper bu nekk
Transformatëri gis-gis ak anam yu bari dañuy leble GeGLU/SwiGLU ngir gëna suqali seen MLP
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the SwiGLU and Gated Activations quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Aktiwisaasioŋ xaymawaat kompromis
Laaj yi ñuy faral di laaj
What is SwiGLU and Gated Activations?
SwiGLU fonction aktivasioŋ la buy yokk benn projection ligneer bu duggal bi ak ñaareelu projection buñ aktive ci Swish, muy liggéey ni buntu buñ mëna jàng, bu aju ci done yi ci biir transformatër biy feed-forward layers. Dafay gëna suqali kalite modelu làkk, moo tax daanaka LLM bu bees bi daf koy jëfandikoo.
Ban jëf core mooy màndargaal ab Unité Lineaire Gated?
GLU dafay yokk benn projection valeur ak benn projection gate ci wàllu élément, suko defee reso bi di doxal xibaar bi ci dimension bu nekk.
Ban fonction activation la SwiGLU di jëfandikoo ci branche gate bi?
SwiGLU dafay jëfandikoo Swish, ñu koy woowe SiLU, ñu koy tekki x * sigmoid (beta * x), ngir bànxaasu buntu bi.
Ñaata matris yu diis la benn bloku feed-forward bu SwiGLU di jëfandikoo?
SwiGLU soxla na ñatti matris: projection buntu bi, projection valeur bi, ak projection biy génn.
Lan moo waral dimension bu nëbbu bi di gëna bari lu tollu ci ñaar ci ñett ci SwiGLU layers?
Ñatteelu matrix bi dina yokk calcul bi, kon wàññi dayo bu nëbbu bi ci lu tollu ci 8/3 d dafay tax budget bi méngoo.
Kan moo siiwal SwiGLU ngir transformatër feed-forward ci 2020?
Notu 2020 bu Noam Shazeer 'GLU Variantes yokk Transformer' dugal SwiGLU te wane njariñam ci jaxasoo.