SwiGLU n'ibikorwa bya Gate
SwiGLU nigikorwa cyo gufunga amarembo agwiza umurongo umwe ugereranya ibyinjijwe na Swish-ikora ya kabiri ya projection, ikora nk'irembo ryiga, rishingiye ku irembo imbere muri transformateur igaburira imbere.
Incamake
It consistently improves language-model quality, which is why nearly every modern LLM uses it.
Kwibira cyane
Ihinduramiterere risanzwe rigaburira-imbere ni imirongo ibiri igizwe na ReLU cyangwa GELU hagati. Gates Linear Units, yatanzwe na Dauphin nabandi. muri 2016, gabanya projection ya mbere mo kabiri hanyuma ukoreshe igice kimwe kugirango winjire ikindi ukoresheje ibintu-bigwira. SwiGLU, ikunzwe na Noam Shazeer muri 2020, ikoresha imikorere ya Swish (SiLU) kuri iryo rembo: ibisohoka = (Swish (xW) * (xV)) W2, hamwe na matrica eshatu z'uburemere aho kuba ebyiri. Irembo ryemerera umuyoboro guhitamo cyangwa guhagarika amakuru kurwego. Kuberako kongeramo matrix ya gatatu ikura ibipimo, gushyira mubikorwa bigabanya ibipimo byihishe hafi bibiri bya gatatu kuburyo compte yose iguma igereranijwe na GELU MLP. Ubushakashatsi bwa Shazeer bwerekanye inyungu zishobora kugerwaho, kandi LLaMA, PaLM, na Mistral bose barabyemeje.
Ubushishozi
Swish ni x * sigmoid (beta * x), imikorere yoroshye, itari monotonic, itandukanye na ReLU, yemerera indangagaciro ntoya binyuze. Muri SwiGLU ishami rya 'irembo' Swish (xW) itanga indangagaciro hafi 0 cyangwa 1 zigwiza 'agaciro' ishami xV element-ifite ubwenge, bityo buri gice cyihishe umusanzu uhindurwa nikimenyetso cyize, cyinjiza-gishingiye. Matrix ya gatatu yuburemere nigiciro; bibiri bya gatatu byihishe-ubunini bwamayeri bituma ingengo yimari ya FLOP ihuye na vanilla ibiryo-byimbere.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Kazoza ka SwiGLU na Gatesi Ibikorwa
SwiGLU yashinze imizi nka MLP isanzwe muburemere bwa LLMs kandi ntibishoboka ko yimurwa vuba. Icyerekezo gifatika kirimo impinduka za GeGLU na ReGLU, zahujwe na GPU intungamubiri zibara ibipimo byombi mumurongo umwe, no guhuza MLP zifunze hamwe nuruvange-rwinzobere kuburyo buri mpuguke ubwayo ari blok ya SwiGLU. Abashakashatsi barimo kwiga kandi impamvu amarembo afasha gutezimbere, bagamije gukora amarembo ahendutse.
Gushyira mu bikorwa Isi
LLaMA, PaLM, na Mistral basimbuza GELU ibiryo-byimbere hamwe na SwiGLU kugirango bagabanye urujijo kuri compte ingana
Ibipimo byihishe bipimirwa kuri bibiri bya gatatu (8/3 d) kugirango matrike yinyongera yinjirira idatera FLOPs
Uruvange-rwinzobere moderi nka Mixtral ikoresha blike ya SwiGLU nkumuyoboro wa buri muhanga ugaburira
Icyerekezo hamwe na multimodal transformateur baguza GeGLU / SwiGLU amarembo kugirango bateze imbere MLP sublayers
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the SwiGLU and Gated Activations quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Igikorwa cyo Kwishura Ibicuruzwa
Ibibazo bikunze kubazwa
What is SwiGLU and Gated Activations?
SwiGLU nigikorwa cyo gufunga amarembo agwiza umurongo umwe ugereranya ibyinjijwe na Swish-ikora ya kabiri ya projection, ikora nk'irembo ryiga, rishingiye ku irembo imbere muri transformateur igaburira imbere. Ihora itezimbere ururimi-rwicyitegererezo, niyo mpamvu hafi ya LLM igezweho ikoresha.
Ni ikihe gikorwa cy'ibanze gisobanura umurongo wa Gateur?
GLU igwiza agaciro projection kumarembo yerekana ibintu-neza, kureka imiyoboro igenzura amakuru atemba murwego.
Nibihe bikorwa byo gukora SwiGLU ikoresha ishami ryayo?
SwiGLU ikoresha Swish, nanone yitwa SiLU, isobanurwa nka x * sigmoid (beta * x), ku ishami ryinjira.
Ni bangahe matrices yuburemere SwiGLU igaburira-imbere ikumira?
SwiGLU ikeneye matrices eshatu: umushinga w irembo, agaciro kerekana, hamwe nibisohoka.
Ni ukubera iki igipimo cyihishe gisanzwe gipima hafi bibiri bya gatatu mubice bya SwiGLU?
Matrix ya gatatu ubundi yazamura compte, bityo kugabanya ubunini bwihishe hafi 8/3 d bituma ingengo yimari ihuye.
Ninde wamamaje SwiGLU ya transfert yo kugaburira imbere muri 2020?
Noam Shazeer inoti ya 2020 'GLU Variants Impinduka Transformer' yerekanye SwiGLU kandi yerekana inyungu zayo zitangaje.