SwiGLU na arụ ọrụ Gated
SwiGLU bụ ọrụ ịgbalite gated nke na-amụba otu ntule ahịrị nke ntinye site na ntule nke abụọ na-arụ ọrụ Swish, na-eme ka ọnụ ụzọ amụta, dabere data n'ime ọkwa ntụgharị ntụgharị.
Nchịkọta
It consistently improves language-model quality, which is why nearly every modern LLM uses it.
Ime miri emi
Ngwongwo ihe ngbanwe ihe ngbanwe ọkọlọtọ bụ akwa ahịrị abụọ nwere ReLU ma ọ bụ GELU n'etiti. Ngalaba Linear Gated, nke Dauphin et al tụpụtara. na 2016, kewaa ntule nke mbụ ụzọ abụọ ma jiri otu ọkara mechie nke ọzọ site na ntinye nke amamihe. SwiGLU, nke Noam Shazeer na-ewu ewu na 2020, na-eji ọrụ Swish (SiLU) maka ọnụ ụzọ ahụ: mmepụta = (Swish(xW) * (xV)) W2, nwere matrices dị arọ atọ kama ịbụ abụọ. Ọnụ ụzọ ahụ na-ahapụ netwọk ahụ ka ọ gafee ma ọ bụ mechie ozi n'otu akụkụ. N'ihi na ịgbakwụnye matriks nke atọ na-eto parampat, mmejuputa iwu na-ebelata akụkụ ahụ zoro ezo ruo ihe dị ka ụzọ abụọ n'ụzọ atọ ka mkpokọta mkpokọta ga-adị ka GELU MLP. Nnwale Shazeer gosipụtara uru mgbagwoju anya enwere ike ịtụte ya, na LLAMA, PaLM na Mistral niile nakweere ya.
Nghọta nka nka
Swish bụ x * sigmoid (beta * x), ọrụ dị nro, anaghị arụ ọrụ nke na-adịghị ka ReLU, na-enye ohere maka obere ụkpụrụ adịghị mma. Na SwiGLU alaka 'ọnụ ụzọ' Swish(xW) na-ewepụta ụkpụrụ dị nso 0 ma ọ bụ 1 na-amụba ngalaba 'uru' xV element-maara ihe, ya mere onyinye nke otu ọ bụla zoro ezo na-eji akara mmụta na-adabere na ntinye. Matrix arọ nke atọ bụ ọnụ ahịa; ụzọ abụọ n'ụzọ atọ zoro ezo-size aghụghọ na-eme ka FLOP mmefu ego kwekọọ na a vanilla ndepụta oyi akwa-ga n'ihu.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke SwiGLU na mmemme Gated
SwiGLU gbanyere mkpọrọgwụ dị ka MLP ndabara na LLM dị arọ mepere emepe ma o yighị ka a ga-ewepụ ya n'oge adịghị anya. Ntuziaka na-arụ ọrụ gụnyere ụdị GeGLU na ReGLU, kernel GPU jikọtara ọnụ nke na-agbakọ amụma abụọ ahụ n'otu ngafe, yana ijikọta MLP gated na ndị ọkachamara ngwakọta nke mere ọkachamara ọ bụla bụ n'onwe ya SwiGLU ngọngọ. Ndị na-eme nchọpụta na-enyochakwa ihe kpatara ịgba chaa chaa na-enyere aka nchacha mma, na-achọ imepụta ọnụ ụzọ ámá dị ọnụ ala karịa.
Mmejuputa n'ezie n'ụwa
LLAMA, PaLM, na Mistral jiri SwiGLU dochie oyi akwa na-aga n'ihu GELU iji wedata mgbagwoju anya n'otu oge.
A na-atụba akụkụ ahụ zoro ezo ruo ihe dị ka ụzọ abụọ n'ụzọ atọ (8/3 d) yabụ matriks gating agbakwunyere anaghị ebute FLOPs.
Ngwakọta-nke-ọkachamara ụdị dị ka Mixtral na-eji SwiGLU blocks dị ka onye ọkachamara na-enyefe netwọkụ.
Ọhụụ na mgbanwe mgbanwe dị iche iche na-agbaziri GeGLU/SwiGLU gating iji melite sublayers MLP ha
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the SwiGLU and Gated Activations quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Azụmahịa mweghachi nke ịgbalite
Ajụjụ a na-ajụkarị
What is SwiGLU and Gated Activations?
SwiGLU bụ ọrụ ịgbalite gated nke na-amụba otu ntule ahịrị nke ntinye site na ntule nke abụọ na-arụ ọrụ Swish, na-eme ka ọnụ ụzọ amụta, dabere data n'ime ọkwa ntụgharị ntụgharị. Ọ na-abawanye ogo ụdị asụsụ, nke mere na ọ fọrọ nke nta ka ọ bụrụ LLM ọgbara ọhụrụ ọ bụla na-eji ya.
Kedu isi ọrụ na-akọwa nkeji Linear Gated?
A GLU na-amụba ntule uru site na ntule ọnụ ụzọ ámá-maara ihe, na-ahapụ ozi njikwa netwọkụ ka ọ na-aga n'otu akụkụ.
Kedu ọrụ ịgbalite SwiGLU na-eji maka alaka ọnụ ụzọ ámá ya?
SwiGLU na-eji Swish, nke a na-akpọ SiLU, akọwara dị ka x * sigmoid(beta*x), maka ngalaba gating.
Kedu matrices arọ ole ka SwiGLU na-egbochi ihe na-aga n'ihu na-eji?
SwiGLU chọrọ matrices atọ: ntule ọnụ ụzọ ámá, ntule uru, na ntule mmepụta.
Kedu ihe kpatara na a na-ejikarị akụkụ ahụ zoro ezo ruo ihe dị ka ụzọ abụọ n'ụzọ atọ na ọkwa SwiGLU?
Matriks nke atọ ga-agbakọ ọnụ ma ọ bụghị ya, ya mere ibelata nha zoro ezo ka ọ bụrụ 8/3 d na-eme ka mmefu ego ahụ kwekọọ.
Kedu onye kwalitere SwiGLU maka ọkwa na-ebugharị ihe ntụgharị na 2020?
Noam Shazeer's 2020 ndetu 'GLU Variants Improve Transformer' webatara SwiGLU wee gosi uru mgbagwoju anya ya.