SwiGLU da Ayyukan Gated
SwiGLU aikin kunnawa mai gate wanda ke ninka tsinkayar tsinkaya guda ɗaya na shigarwar ta hanyar tsinkayar Swish da aka kunna ta biyu, tana aiki azaman abin koyo, kofa mai dogaro da bayanai a cikin yadudduka ciyar da taswira.
Dubawa
Yana ci gaba da haɓaka ingancin ƙirar harshe, wanda shine dalilin da ya sa kusan kowane LLM na zamani ke amfani da shi.
Zurfafa nutsewa
Madaidaicin tsarin ciyarwa na gaba shine yadudduka na layi biyu tare da ReLU ko GELU a tsakani. Raka'a Linear Gated, Dauphin et al. a cikin 2016, raba tsinkayar farko zuwa rabi biyu kuma yi amfani da rabi don kori ɗayan ta hanyar haɓaka-hikima. SwiGLU, wanda Noam Shazeer ya shahara a cikin 2020, yana amfani da aikin Swish (SiLU) don waccan ƙofar: fitarwa = (Swish(xW) * (xV)) W2, tare da matrices masu nauyi guda uku maimakon biyu. Gating ɗin yana ƙyale cibiyar sadarwar zaɓin wucewa ko murkushe bayanai kowane girma. Saboda ƙara matrix na uku yana haɓaka sigogi, aiwatarwa yana rage girman ɓoyayyiyar zuwa kusan kashi biyu bisa uku don haka jimlar ƙididdigewa ta kasance daidai da GELU MLP. Gwaje-gwajen Shazeer sun nuna ribar ruɗani da ake iya aunawa, kuma LLAMA, PaLM, da Mistral duk sun karɓe ta.
Fahimtar Fasaha
Swish shine x * sigmoid(beta*x), santsi, aiki mara monotonic wanda, sabanin ReLU, yana ba da damar ƙananan ƙima mara kyau ta hanyar. A cikin SwiGLU reshen 'ƙofa' Swish(xW) yana samar da ƙima kusa da 0 ko 1 waɗanda ke ninka 'daraja' reshe xV element-hikima, don haka kowane ɗayan ɓoyayyiyar gudummawar tana canzawa ta hanyar koyo, siginar dogaro da shigarwa. Matrix nauyi na uku shine farashi; dabarar girman ɓoyayye kashi biyu cikin uku yana kiyaye kasafin kuɗi na FLOP daidai da Layer ciyar da gaba.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar SwiGLU da Ayyukan Gated
SwiGLU yana da tushe a matsayin tsoho MLP a cikin LLMs masu nauyi mai buɗewa kuma ba zai yuwu a ƙaura ba nan ba da jimawa ba. Hanyoyi masu aiki sun haɗa da bambance-bambancen GeGLU da ReGLU, kernels GPU masu haɗaka waɗanda ke ƙididdige tsinkayar duka biyun a cikin fasfo ɗaya, da haɗa MLPs gated tare da ƙwararrun masana don haka kowane ƙwararre kansa shine toshe SwiGLU. Masu bincike kuma suna nazarin dalilin da yasa gating ke taimakawa ingantawa, da nufin zayyana ko da kofofin rahusa.
Aiwatar da Gaskiyar Duniya
LLAMA, PaLM, da Mistral sun maye gurbin GELU ciyarwar gaba Layer tare da SwiGLU don rage damuwa a daidai lissafi.
An daidaita girman ɓoyayyiyar zuwa kusan kashi biyu bisa uku (8/3 d) don haka ƙarin matrix ɗin gating baya haɓaka FLOPs.
Samfuran ƙwararrun masana kamar Mixtral suna amfani da tubalan SwiGLU azaman hanyar sadarwar ciyarwar kowane ƙwararru.
Hanyoyi da masu canji na multimodal suna aro GeGLU/SwiGLU gating don inganta masu sa hannun MLP su
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the SwiGLU and Gated Activations quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Kasuwancin sake lissafin kunnawa
Tambayoyin da ake yawan yi
Menene SwiGLU da Ayyukan Gated?
SwiGLU aikin kunnawa mai gate wanda ke ninka tsinkayar tsinkaya guda ɗaya na shigarwar ta hanyar tsinkayar Swish da aka kunna ta biyu, tana aiki azaman abin koyo, kofa mai dogaro da bayanai a cikin yadudduka ciyar da taswira. Yana ci gaba da haɓaka ingancin ƙirar harshe, wanda shine dalilin da ya sa kusan kowane LLM na zamani ke amfani da shi.
Wanne ainihin aiki ne ke ma'anar Rukunin Linear Gated?
A GLU yana ninka tsinkayar ƙima ta hanyar tsinkayar ƙofa-hikima, yana barin bayanan sarrafa cibiyar sadarwa ya gudana kowane girma.
Wane aikin kunnawa SwiGLU ke amfani dashi don reshen ƙofa?
SwiGLU yana amfani da Swish, wanda kuma ake kira SiLU, wanda aka ayyana da x * sigmoid(beta*x), don reshen gating.
Nawa matrices masu nauyi ke amfani da toshe-gaba-gaba na SwiGLU?
SwiGLU yana buƙatar matrices guda uku: tsinkayar ƙofar, tsinkayar ƙima, da tsinkayen fitarwa.
Me yasa yawan ɓoyayyun girman ke ƙanƙanta zuwa kusan kashi biyu bisa uku a cikin yadudduka na SwiGLU?
Matrix na uku in ba haka ba zai haifar da ƙididdigewa, don haka rage girman ɓoye zuwa kusan 8/3 d yana kiyaye kasafin kuɗi daidai.
Wanene ya shahara da SwiGLU don yaduddukan ciyarwa na gaba a cikin 2020?
Noam Shazeer's 2020 bayanin kula 'GLU Variants Inganta Mai Canjawa' ya gabatar da SwiGLU kuma ya nuna ribar ruɗaninsa.