Dubawa
SwiGLU aikin kunnawa mai gate wanda ke ninka tsinkayar tsinkaya guda ɗaya na shigarwar ta hanyar tsinkayar Swish da aka kunna ta biyu, tana aiki azaman abin koyo, kofa mai dogaro da bayanai a cikin yadudduka ciyar da taswira. Yana ci gaba da haɓaka ingancin ƙirar harshe, wanda shine dalilin da ya sa kusan kowane LLM na zamani ke amfani da shi.
SwiGLU da Gated Activations wani shingen gini ne na fasaha wanda ke shafar ingancin samfuri, farashin kayayyakin more rayuwa, latency, da aminci a sikeli.
Zurfafa nutsewa
Madaidaicin tsarin ciyarwa na gaba shine yadudduka na layi biyu tare da ReLU ko GELU a tsakani. Raka'a Linear Gated, Dauphin et al. a cikin 2016, raba tsinkayar farko zuwa rabi biyu kuma yi amfani da rabi don kori ɗayan ta hanyar haɓaka-hikima. SwiGLU, wanda Noam Shazeer ya shahara a cikin 2020, yana amfani da aikin Swish (SiLU) don waccan ƙofar: fitarwa = (Swish(xW) * (xV)) W2, tare da matrices masu nauyi guda uku maimakon biyu. Gating ɗin yana ƙyale cibiyar sadarwar zaɓin wucewa ko murkushe bayanai kowane girma. Saboda ƙara matrix na uku yana haɓaka sigogi, aiwatarwa yana rage girman ɓoyayyiyar zuwa kusan kashi biyu bisa uku don haka jimlar ƙididdigewa ta kasance daidai da GELU MLP. Gwaje-gwajen Shazeer sun nuna ribar ruɗani da ake iya aunawa, kuma LLAMA, PaLM, da Mistral duk sun karɓe ta.
Fahimtar Fasaha
Swish shine x * sigmoid(beta*x), santsi, aiki mara monotonic wanda, sabanin ReLU, yana ba da damar ƙananan ƙima mara kyau ta hanyar. A cikin SwiGLU reshen 'ƙofa' Swish(xW) yana samar da ƙima kusa da 0 ko 1 waɗanda ke ninka 'daraja' reshe xV element-hikima, don haka kowane ɗayan ɓoyayyiyar gudummawar tana canzawa ta hanyar koyo, siginar dogaro da shigarwa. Matrix nauyi na uku shine farashi; dabarar girman ɓoyayye kashi biyu cikin uku yana kiyaye kasafin kuɗi na FLOP daidai da Layer ciyar da gaba.
Jagorar SwiGLU da Ayyukan Gated
SwiGLU aikin kunnawa mai gate wanda ke ninka tsinkayar tsinkaya guda ɗaya na shigarwar ta hanyar tsinkayar Swish da aka kunna ta biyu, tana aiki azaman abin koyo, kofa mai dogaro da bayanai a cikin yadudduka ciyar da taswira. Yana ci gaba da haɓaka ingancin ƙirar harshe, wanda shine dalilin da ya sa kusan kowane LLM na zamani ke amfani da shi. SwiGLU da Gated Activations wani shingen gini ne na fasaha wanda ke shafar ingancin samfuri, farashin kayayyakin more rayuwa, latency, da aminci a sikeli. Don gina zurfin fahimta, bi SwiGLU da Gated Activations a matsayin samfurin aiki, ba sifa ɗaya ba: ayyana sakamakon da ake so, fayyace zato, da raba abin da tsarin zai iya yi da dogaro daga abin da har yanzu yana buƙatar yanke hukunci na ƙwararru.
A aikace, ƙungiyoyi masu ƙarfi masu amfani da SwiGLU da Gated Activations suna haɓaka gine-gine, bayanai, da zaɓin abubuwan more rayuwa akan dogaro da farashi. Suna rubuta ƙayyadaddun ƙa'idodin nasara, gwaji akan bayanan gaskiya da gudanawar aiki, da jujjuyawar bisa ga tsarin gazawar da aka lura maimakon cin nasara na lokaci ɗaya. Wannan shine inda fahimtar ka'idar ta juya zuwa iyawa mai dorewa a cikin samfura, manufofi, da ayyuka.
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru. A lokaci guda, Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin. Hanyar da ta fi dacewa ita ce haɗa saurin gwaji tare da horon gudanarwa: gudanar da matukin jirgi, kama shaida, buga rajistan ayyukan yanke shawara, da ci gaba da sabunta abubuwan tsaro kamar yadda halayen ƙira, tsammanin mai amfani, da buƙatun tsari ke tasowa.
Dabarun Tasiri
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru. A cikin ƙawance masu inganci, ana fassara wannan zuwa ƙa'idodin aiki waɗanda za a iya aunawa, iyakokin ikon mallaka, da kuma bita-da-kullin bita don ƙungiyoyi su iya haɓaka kwarin gwiwa a maimakon ɓata shakku.
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba. A cikin ƙawance masu inganci, ana fassara wannan zuwa ƙa'idodin aiki waɗanda za a iya aunawa, iyakokin ikon mallaka, da kuma bita-da-kullin bita don ƙungiyoyi su iya haɓaka kwarin gwiwa a maimakon ɓata shakku.
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa. A cikin ƙawance masu inganci, ana fassara wannan zuwa ƙa'idodin aiki waɗanda za a iya aunawa, iyakokin ikon mallaka, da kuma bita-da-kullin bita don ƙungiyoyi su iya haɓaka kwarin gwiwa a maimakon ɓata shakku.
Aiwatar da Gaskiyar Duniya
LLAMA, PaLM, da Mistral sun maye gurbin GELU ciyarwar gaba Layer tare da SwiGLU don rage damuwa a daidai lissafi.
An daidaita girman ɓoyayyiyar zuwa kusan kashi biyu bisa uku (8/3 d) don haka ƙarin matrix ɗin gating baya haɓaka FLOPs.
Samfuran ƙwararrun masana kamar Mixtral suna amfani da tubalan SwiGLU azaman hanyar sadarwar ciyarwar kowane ƙwararru.
Hanyoyi da masu canji na multimodal suna aro GeGLU/SwiGLU gating don inganta masu sa hannun MLP su
Hanyoyin Aiwatarwa
SwiGLU da Gated Activations a aikace
LLAMA, PaLM, da Mistral sun maye gurbin GELU ciyarwar gaba Layer tare da SwiGLU zuwa rage damuwa a daidai lissafi.
LLAMA, PaLM, da Mistral sun maye gurbin GELU ciyarwar gaba tare da SwiGLU don rage damuwa a daidaitattun ƙididdiga Ƙungiyoyi yawanci suna samun sakamako mafi kyau lokacin da suka ayyana ma'auni masu inganci a gaba, kiyaye hanyar haɓakar ɗan adam don shari'o'in gefe, da bin duk abubuwan da ake samu da kuma farashi na kuskure akan lokaci.
SwiGLU da Gated Activations a aikace
An ƙididdige girman ɓoyayyiyar zuwa kusan kashi biyu bisa uku (8/3 d) don haka ƙarin matrix ɗin gating baya haɓaka FLOPs.
An ƙaddamar da girman ɓoyayyiyar zuwa kusan kashi biyu bisa uku (8/3 d) don haka ƙarin matrix ɗin gating ba ya haɓaka FLOPs Ƙungiyoyi yawanci suna samun sakamako mafi kyau lokacin da suka ayyana ma'auni masu inganci a gaba, kiyaye hanyar haɓakar ɗan adam don ƙararraki, da bin diddigin duk abubuwan da aka samu da ƙimar kuɗi akan lokaci.
SwiGLU da Gated Activations a aikace
Samfuran ƙwararrun masana kamar Mixtral suna amfani da tubalan SwiGLU azaman hanyar sadarwar ciyarwar kowane gwani.
Samfuran ƙwararrun masana kamar Mixtral suna amfani da shingen SwiGLU kamar yadda ƙwararrun ƙwararrun ƙwararrun ƙwararrun ƙwararrun ƙwararrun ƙwararrun ƙwararrun ƙwararrun ƙwararrun ƙwararrun ƙwararrun ƙwararrun ƙwararrun ƙwararrun ƙwararrun ƙwararru yawanci suna samun sakamako mafi kyau lokacin da suke ayyana ƙofofin inganci a gaba, kiyaye hanyar haɓaka ɗan adam don shari'o'in gefe, da bin diddigin nasarorin samarwa da ƙimar kuskure akan lokaci.
SwiGLU da Gated Activations a aikace
Hanyoyi da masu canji na multimodal suna aro GeGLU/SwiGLU gating don inganta masu sa hannun MLP su.
Hanyoyi da masu canji na multimodal suna aro GeGLU/SwiGLU gating don haɓaka masu biyan kuɗi na MLP Ƙungiyoyi yawanci suna samun sakamako mafi kyau lokacin da suka ayyana ƙofofin inganci a gaba, kiyaye hanyar haɓakar ɗan adam don shari'o'i, da bin duk nasarorin samarwa da ƙimar kuskure akan lokaci.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa. Ɗauki kowane mataki azaman ƙofar shaida: idan ba a cika sharuɗɗa ba, dakatar da fitar, rufe tazarar, sannan kawai faɗaɗa amfani.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai. Ɗauki kowane mataki azaman ƙofar shaida: idan ba a cika sharuɗɗa ba, dakatar da fitar, rufe tazarar, sannan kawai faɗaɗa amfani.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani. Ɗauki kowane mataki azaman ƙofar shaida: idan ba a cika sharuɗɗa ba, dakatar da fitar, rufe tazarar, sannan kawai faɗaɗa amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli. Ɗauki kowane mataki azaman ƙofar shaida: idan ba a cika sharuɗɗa ba, dakatar da fitar, rufe tazarar, sannan kawai faɗaɗa amfani.