SwiGLU iyo Dhaqdhaqaaqyada Gated
SwiGLU waa hawl firfircooni dabin ah oo ku dhufata hal saadaasha toosan ee gelinta iyadoo la raacayo saadaasha Swish-firfircoon ee labaad, una dhaqanta sidii mid la baran karo, irid xog ku tiirsan oo gudaha lakabyada gudbinta beddelka ah.
Dulmar
It consistently improves language-model quality, which is why nearly every modern LLM uses it.
quusid qoto dheer
Bloog-horumar-is-beddelka caadiga ah waa laba lakab oo toosan oo leh ReLU ama GELU inta u dhaxaysa. Unugyada toosan ee Gated, waxaa soo jeediyay Dauphin et al. sanadka 2016, u kala qaybi saadaasha kowaad laba qaybood oo isticmaal badh si aad kan kale uga soo gasho iyada oo la adeegsanayo isku dhufashada element-wise. SwiGLU, oo uu caanka ku yahay Noam Shazeer sanadka 2020, ayaa u adeegsata shaqada Swish (SiLU) albaabkaas: wax soo saarka = (Swish(xW) * (xV)) W2, oo leh saddex mitir oo miisaan halkii laba. Gating-ku wuxuu u ogolaanayaa shabakadu inay si xushmad leh u gudbiso ama u cabudhiso xogta dhinac kasta. Sababtoo ah ku daridda shaxanka saddexaad waxay koraan cabbirrada, fulinta waxay hoos u dhigtaa cabbirka qarsoon ilaa qiyaastii saddex-meelood laba meel sidaas darteed wadarta xisaabinta waxay u dhigantaa GELU MLP. Tijaabooyinkii Shazeer waxay muujiyeen faa'iidooyin jahawareer la qiyaasi karo, LLAMA, PaLM, iyo Mistral dhamaantood way qaateen.
Aragtida Farsamada
Swish waa x * sigmoid(beta*x), siman, shaqo aan monotonic ahayn, taas oo ka duwan ReLU, u ogolaata qiyam yar oo taban. Gudaha SwiGLU laanta 'albaabka' Swish(xW) waxay soo saartaa qiyam ku dhow 0 ama 1 taaso ku dhufata laanta 'qiimaha' xV element-wise, sidaa darteed qayb kasta oo dahsoon wax ku biirinta waxaa habeeya calaamad la bartay, ku-tiirsanaan. Matrixka miisaanka saddexaad waa kharashka; saddex-meelood laba meelood khiyaanada cabbirka qarsoon waxay ku ilaalinaysaa miisaaniyadda FLOP inay la mid noqoto lakabka hore ee quudinta vanilj.
Saamaynta Istiraatijiyadeed
Qiimaha iyo miisaaniyada
Go'aamada qaab-dhismeedku waxay horseedaan waxqabadka iyo kharashka hawlgalka sannadaha.
Go'aamo cad
Waxbarashada farsamada waxay ka caawisaa kooxaha inay doortaan xidhmo sax ah, ma aha oo kaliya kan ugu cusub.
Xakamaynta tayada
Doorashooyinka injineernimada ee wanaagsan waxay yareeyaan shilalka la isku halleyn karo ee wax soo saarka.
Mustaqbalka SwiGLU iyo Hawlaha Gated
SwiGLU waxay ku qotontaa sidii MLP-ga caadiga ah ee LLM-yada miisaanka-furan oo uma badna in dhawaan la barokiciyo. Tilmaamaha firfircoon waxaa ka mid ah noocyada GeGLU iyo ReGLU, kernels GPU isku dhafan oo ku xisaabiya labadaba saadaasha hal baas, iyo isku darka MLP-yada oo leh khubaro isku dhafan si khabiir kastaa laftiisa u yahay SwiGLU block. Cilmi-baarayaashu waxay sidoo kale baranayaan sababta gatinku u caawiyo hagaajinta, iyaga oo ujeedadoodu tahay inay naqshadeeyaan xitaa albaabo ka jaban.
Dhaqangelinta Adduunka-dhabta ah
LLAMA, PaLM, iyo Mistral waxay ku beddelaan lakabka hore ee quudinta GELU ee SwiGLU si ay hoos ugu dhigaan jaahwareerka isla xisaabtan
Cabbirka dahsoon ayaa la miisaamay ilaa saddex meelood laba meel (8/3 d) markaa matrixka dheeriga ah ma buufiyo FLOPs
Moodooyinka isku dhafka ah ee khubarada ah sida Mixtral waxay isticmaalaan SwiGLU blocks sida shabakadda quudinta khabiirka
Aragtida iyo transformers multimodal waxay amaahdaan GeGLU/SwiGLU gating si ay u horumariyaan qaybiyayaalkooda MLP
Khatarta & Dariiqyada Ilaalada
Hagaajinta hal bartilmaameed waxay qarin kartaa daciifnimada nidaamka ballaaran.
Kaabayaasha dhaqaalaha iyo dayactirka inta badan waa la dhayalsadaa.
Nabadgelyada iyo daldaloolada u fiirsashada ayaa kori kara marka nidaamyadu noqdaan kuwo aad u adag.
Qorshe Hawleedka Dhaqangelinta
Qeex daahida, tayada, iyo bartilmaameedyada qiimaha ka hor inta aan la hirgelin.
Benchmark marka la eego culeyska dhabta ah iyo xaaladaha xogta.
La socodka qalabka khaladaadka, leexashada, iyo saamaynta isticmaalaha.
U diyaari dib-u-noqoshada iyo dariiqyada jawaab-celinta dhacdada ka hor inta aanad miisaan.
Sii wad Sahaminta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the SwiGLU and Gated Activations quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Hagaha xiga
Waxyeelada dib u xisaabinta firfircoonida
Su'aalaha soo noqnoqda
What is SwiGLU and Gated Activations?
SwiGLU waa hawl firfircooni dabin ah oo ku dhufata hal saadaasha toosan ee gelinta iyadoo la raacayo saadaasha Swish-firfircoon ee labaad, una dhaqanta sidii mid la baran karo, irid xog ku tiirsan oo gudaha lakabyada gudbinta beddelka ah. Waxay si joogto ah u wanaajisaa tayada nooca luqadda, taas oo ah sababta ku dhawaad kasta oo casri ah LLM u isticmaalo.
Hawlgalkee xudunta u ah ayaa qeexaya Unugga toosan ee Gated?
GLU waxay ku dhufataa saadaasha qiimaha iyada oo loo eegayo qaybta saadaasha albaabka, iyada oo u oggolaanaysa shabakaddu inay maamusho macluumaadka socodka cabbir kasta.
Shaqadee hawl qabad ayay SwiGLU u isticmaashaa laanteeda albaabka?
SwiGLU waxay isticmaashaa Swish, sidoo kale loo yaqaan SiLU, oo lagu qeexay x * sigmoid(beta*x), laanta gating.
Immisa xabbo oo miisaan ah ayaa SwiGLU quudin-horumarintu isticmaashaa?
SwiGLU waxay u baahan tahay saddex mitir: saadaasha albaabka, saadaasha qiimaha, iyo saadaasha wax soo saarka.
Waa maxay sababta cabbirka dahsoon sida caadiga ah loogu qiyaaso saddex-meeloodlaba laba lakabyada SwiGLU?
Matrixka saddexaad wuxuu si kale u buufinayaa xisaabinta, markaa hoos u dhigista cabbirka qarsoon ilaa 8/3 d waxay ilaalinaysaa miisaaniyaddu inay iswaafaqdo.
Yaa ku caan baxay SwiGLU lakabyada quudinta gudbinta beddelka sanadka 2020?
Noam Shazeer's 2020 note 'GLU Variants Hagaajinta Transformer' waxay soo bandhigtay SwiGLU waxayna muujisay faa'iidooyinkeeda wareersan.