I-SwiGLU kanye ne-Gated Activations
I-SwiGLU iwumsebenzi wokwenza kusebenze onesango ophindaphinda ukuqagela okukodwa komugqa kokufakwayo ngokuqagela kwesibili okusetshenziselwe i-Swish, okusebenza njengesango elifundekayo, elincike kudatha ngaphakathi kwezendlalelo zokudlulisela phambili okuphakelayo kwe-transformer.
Uhlolojikelele
It consistently improves language-model quality, which is why nearly every modern LLM uses it.
I-Deep Dive
Ibhulokhi yokuphakela phambili kwe-transformer ejwayelekile iyizendlalelo ezimbili eziqondile ezine-ReLU noma i-GELU phakathi. I-Gated Linear Units, ehlongozwe ngu-Dauphin et al. ngo-2016, hlukanisa i-projection yokuqala ibe izingxenye ezimbili futhi usebenzise ingxenye eyodwa ukuze isango enye ngokuphindaphinda okuhlakaniphile. I-SwiGLU, eyaduma nguNoam Shazeer ngo-2020, isebenzisa umsebenzi we-Swish (SiLU) walelo sango: okukhiphayo = (Swish(xW) * (xV)) W2, enamatrices amathathu esisindo esikhundleni samabili. Isango lokungena livumela inethiwekhi ngokukhetha ukuthi idlule noma icindezele ulwazi ngosayizi ngamunye. Ngenxa yokuthi ukungeza i-matrix yesithathu kukhulisa amapharamitha, ukusetshenziswa kufinyeza ubukhulu obufihliwe ukuze bucishe kube izingxenye ezimbili kwezintathu ukuze ikhompuyutha ephelele ihlale iqhathaniswa ne-GELU MLP. Ukuhlola kuka-Shazeer kubonise izinzuzo ezilinganisekayo zokudideka, futhi i-LLaMA, i-PaLM, ne-Mistral zonke zakwamukela.
I-Technical Insight
I-Swish ithi x * sigmoid(beta*x), umsebenzi obushelelezi, ongewona we-monotonic, ngokungafani ne-ReLU, ovumela amanani amancane angalungile ngawo. Ku-SwiGLU igatsha 'lesango' i-Swish(xW) ikhiqiza amanani aseduze no-0 noma 1 aphindaphinda ingxenye 'yenani' yegatsha xV, ngakho umnikelo weyunithi ngayinye efihliwe ulungiswa ngesignali efundiwe, encike kokufakiwe. I-matrix yesisindo sesithathu yizindleko; iqhinga lokubili kokuthathu losayizi ofihliwe ligcina ibhajethi ye-FLOP ifaniswe nesendlalelo sokuphakela phambili kwe-vanilla.
I-Strategic Impact
Izindleko kanye nesabelomali
Izinqumo zezakhiwo ziqhuba ukusebenza kanye nezindleko zokusebenza iminyaka.
Izinqumo ezicacile
Imfundo yobuchwepheshe isiza amaqembu ukuthi akhethe isitaki esifanele, hhayi nje esisha.
Ukulawulwa kwekhwalithi
Izinketho ezingcono zobunjiniyela zinciphisa izehlakalo ezinokwethenjelwa ekukhiqizeni.
Ikusasa le-SwiGLU kanye ne-Gated Activations
I-SwiGLU igxilile njenge-MLP ezenzakalelayo kuma-LLM anesisindo esivulekile futhi cishe ngeke isuswe maduzane. Izikhombisi-ndlela ezisebenzayo zifaka okuhlukile kwe-GeGLU kanye ne-ReGLU, izikhwebu ze-GPU ezihlanganisiwe ezihlanganisa kokubili ukuqagela ngephasi eyodwa, kanye nokuhlanganisa ama-MLP anamasango nengxube yochwepheshe ukuze uchwepheshe ngamunye abe yibhulokhi ye-SwiGLU. Abacwaningi futhi bayacwaninga ukuthi kungani i-gating isiza ukwenza kahle, kuhloswe ngayo ukuklama ngisho namasango ashibhile.
Ukuqaliswa Komhlaba Wangempela
I-LLaMA, i-PaLM, ne-Mistral ibuyisela isendlalelo sokuphakela phambili se-GELU nge-SwiGLU ukuze kwehle ukudideka ngekhompyutha elinganayo.
Ubukhulu obufihliwe bulinganiswa cishe kuzingxenye ezimbili kwezintathu (8/3 d) ukuze i-matrix eyengeziwe yokufaka isango ingavutheli ama-FLOP
Ingxubevange yamamodeli afana ne-Mixtral asebenzisa i-SwiGLU ivimba njengenethiwekhi yokuphakela phambili kochwepheshe
Umbono kanye nama-multimodal transformer aboleka i-GeGLU/SwiGLU gating ukuze bathuthukise ama-sublayers abo e-MLP
Izingozi & Guardrails
Ukuthuthukisa ibhentshimakhi eyodwa kungafihla ubuthakathaka obubanzi besistimu.
Izindleko zengqalasizinda nezokulungisa zivame ukubukelwa phansi.
Izikhala zokuphepha nokubonakala zingakhula njengoba izinhlelo ziba nzima kakhulu.
Ukuqalisa Umhlahlandlela
Chaza ukubambezeleka, ikhwalithi, nezindleko ezihlosiwe ngaphambi kokuqaliswa.
Ibhentshimakhi ngaphansi komthwalo wangempela nezimo zedatha.
Ukuqapha amathuluzi amaphutha, ukukhukhuleka, nomthelela wabasebenzisi.
Lungiselela izindlela zokuhlehlisa nezigameko ngaphambi kokukala.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the SwiGLU and Gated Activations quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
I-activation Recomputation Tradeoffs
Imibuzo evame ukubuzwa
What is SwiGLU and Gated Activations?
I-SwiGLU iwumsebenzi wokwenza kusebenze onesango ophindaphinda ukuqagela okukodwa komugqa kokufakwayo ngokuqagela kwesibili okusetshenziselwe i-Swish, okusebenza njengesango elifundekayo, elincike kudatha ngaphakathi kwezendlalelo zokudlulisela phambili okuphakelayo kwe-transformer. Ihlala ithuthukisa ikhwalithi yemodeli yolimi, yingakho cishe yonke i-LLM yesimanje iyisebenzisa.
Yimuphi umsebenzi owumongo ochaza i-Gated Linear Unit?
I-GLU iphindaphinda ukuqagela kwevelu nge-elementi yokuqagela kwesango, ivumela inethiwekhi ilawule ukugeleza kolwazi ngohlangothi ngalunye.
Imuphi umsebenzi wokwenza kusebenze i-SwiGLU ewusebenzisela igatsha layo lesango?
I-SwiGLU isebenzisa i-Swish, ebizwa nangokuthi i-SiLU, echazwa ngokuthi x * sigmoid(beta*x), egatsheni lokungena.
Bangaki omatikuletsheni besisindo abasetshenziswa ibhulokhi ye-SwiGLU feed-forward?
I-SwiGLU idinga omatikuletsheni abathathu: ukuqagela kwesango, ukuqagela kwenani, kanye nokuqagela kokuphumayo.
Kungani ubukhulu obufihliwe buvame ukukalwa cishe kuzingxenye ezimbili kwezintathu ngezandlalelo ze-SwiGLU?
I-matrix yesithathu ingase ikhulise ukubala, ngakho-ke ukunciphisa usayizi ofihliwe ukuze cishe u-8/3 d kugcina isabelomali sifaniswa.
Ubani owaduma i-SwiGLU yezendlalelo zokudluliswa kwe-transformer ngo-2020?
Inothi likaNoam Shazeer lika-2020 elithi 'GLU Variants Improve Transformer' yethule i-SwiGLU futhi yabonisa izinzuzo zayo zokudideka.