Overview
SwiGLU igated activation function iyo inowedzera imwe mutsara fungidziro yekupinza neSwish-activated yechipiri fungidziro, ichiita senge inodzidzika, data-rinovimba gedhi mukati metransformer feed-forward layers. Iyo inogara ichinatsiridza mutauro-modhi yemhando, ndosaka ingangoita yese yemazuva ano LLM inoishandisa.
SwiGLU uye Gated Activations chivakwa chehunyanzvi chinokanganisa mhando yemhando, mutengo wezvivakwa, latency, uye kuvimbika pachiyero.
Deep Dive
A standard transformer feed-forward block ane maviri mutsara akaturikidzana ane ReLU kana GELU pakati. Gated Linear Units, yakakurudzirwa naDauphin et al. muna 2016, patsanura yekutanga fungidziro kuita hafu mbiri uye shandisa imwe hafu kugedhi imwe kuburikidza ne element-hungwaru kuwanda. SwiGLU, yakakurumbira naNoam Shazeer muna 2020, inoshandisa iyo Swish (SiLU) basa regedhi iro: kubuda = (Swish(xW) * (xV)) W2, ine huremu hwematiriki matatu panzvimbo pembiri. Iyo gedhi inobvumira network kuti ipfuure kana kudzvanyirira ruzivo pachiyero. Nekuti kuwedzera iyo yechitatu matrix inokura paramita, mashandisirwo anodzikisa yakavanzika dimension kusvika pazvikamu zviviri muzvitatu saka yakazara compute inogara yakafanana neGELU MLP. Kuedza kwaShazeer kwakaratidza kuomerwa kunowanikwa, uye LLaMA, PaLM, uye Mistral vese vakazvigamuchira.
Technical Insight
Swish is x * sigmoid(beta*x), yakatsetseka, isiri-monotonic basa iro, kusiyana neReLU, rinobvumira diki diki hunhu kuburikidza. MuSwiGLU iyo 'gedhi' bazi Swish(xW) inogadzira kukosha padyo ne0 kana 1 iyo inowedzera iyo 'value' bazi xV chinhu-huchenjeri, saka imwe neimwe yakavanzika mupiro wechikamu chinogadziridzwa neyakadzidziswa, inotsamira-chiratidzo chiratidzo. Yechitatu uremu matrix ndiyo mutengo; iwo maviri-matatu akavanzika-saizi trick inochengeta iyo FLOP bhajeti inoenderana nevhavha yekudyisa-mberi layer.
Kubata SwiGLU uye Gated Activations
SwiGLU igated activation function iyo inowedzera imwe mutsara fungidziro yekupinza neSwish-activated yechipiri fungidziro, ichiita senge inodzidzika, data-rinovimba gedhi mukati metransformer feed-forward layers. Iyo inogara ichinatsiridza mutauro-modhi yemhando, ndosaka ingangoita yese yemazuva ano LLM inoishandisa. SwiGLU uye Gated Activations chivakwa chehunyanzvi chinokanganisa mhando yemhando, mutengo wezvivakwa, latency, uye kuvimbika pachiyero. Kuti uvake kunzwisisa kwakadzama, bata SwiGLU uye Gated Activations semuenzaniso wekushandisa, kwete chinhu chimwe chete: tsanangura zvaunoda mhedzisiro, kujekesa fungidziro, uye patsanura izvo zvinogona kuitwa nehurongwa hwakavimbika kubva kune izvo zvichiri kuda kutonga kwenyanzvi.
Mukuita, zvikwata zvakasimba zvinoshandisa SwiGLU uye Gated Activations inokwidziridza zvivakwa, data, uye sarudzo dzezvivakwa zvinopesana nekuvimbika uye mutengo. Ivo vanonyora zvakajeka maitiro ebudiriro, bvunzo vachipokana ne data rechokwadi uye mafambiro ebasa, uye iterate zvichibva pane zvakacherechedzwa maitiro ekutadza kwete kuhwina-nguva imwe chete yebhenji. Apa ndipo apo kunzwisisa kwe theoretical kunoshanduka kuve kugona kwakasimba pane chigadzirwa, mutemo, uye mashandiro.
Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore. Panguva imwecheteyo, Kukwirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba. Nzira yakatsiga ndeyekubatanidza kukurumidza kuyedza nekutonga: mhanyisa vatyairi vendege, tora humbowo, buritsa matanda esarudzo, uye urambe uchivandudza chengetedzo semaitiro emuenzaniso, zvinotarisirwa nemushandisi, uye zvinodikanwa zvekutonga.
Strategic Impact
Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore.
Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete.
Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira.
Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Real-World Implementation
LLaMA, PaLM, uye Mistral inotsiva iyo GELU feed-forward layer ne SwiGLU kudzikisa kuvhiringidzika pakuenzanisa compute.
Iyo yakavanzwa dimension yakayerwa kusvika pazvikamu zviviri muzvitatu (8/3 d) saka iyo yekuwedzera gating matrix haizadze FLOPs.
Musanganiswa-we-nyanzvi modhi dzakadai seMixtral shandisa SwiGLU inovhara seye-nyanzvi yekudyisa-mberi network.
Vision uye multimodal transformers inokwereta GeGLU/SwiGLU gating kuti vavandudze avo MLP sublayers.
Maitiro Ekuita
SwiGLU uye Gated Activation mukuita
LLaMA, PaLM, uye Mistral inotsiva iyo GELU feed-forward layer neSwiGLU kuderedza kuvhiringidzika pakuenzanisa compute.
LLaMA, PaLM, uye Mistral inotsiva iyo GELU feed-forward layer neSwiGLU kuti ideredze kuvhiringidzika pakuenzanisa komputa Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi ekumucheto, uye kuteedzera zvese zvakawanikwa zvechigadzirwa uye kukanganisa mutengo nekufamba kwenguva.
SwiGLU uye Gated Activation mukuita
Iyo yakavanzwa dimension inoyerwa kusvika pazvikamu zviviri muzvitatu (8/3 d) saka iyo yekuwedzera gating matrix haizadze FLOPs.
Iyo yakavanzika dimension yakayerwa kusvika pazvikamu zviviri muzvitatu (8/3 d) saka iyo yekuwedzera gating matrix haiputire FLOPs Matimu anowanzo kuwana mibairo iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi ekumucheto, uye kuteedzera zvese zvakawanikwa zvechigadzirwa uye kukanganisa mutengo nekufamba kwenguva.
SwiGLU uye Gated Activation mukuita
Musanganiswa-we-nyanzvi modhi dzakadai seMixtral shandisa SwiGLU inovharira seye-nyanzvi yekudyisa-mberi network.
Musanganiswa-we-nyanzvi modhi dzakadai seMixtral shandisa SwiGLU inovharira seye-nyanzvi yekudyisa-mberi network Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi ekumucheto, uye kuteedzera zvese zvakawanikwa zvechigadzirwa uye kukanganisa mutengo nekufamba kwenguva.
SwiGLU uye Gated Activation mukuita
Vision uye multimodal transformers inokwereta GeGLU/SwiGLU gating kuti vavandudze avo MLP sublayers.
Vision uye multimodal transformers inokwereta GeGLU/SwiGLU gating kuti vavandudze avo MLP sublayers Matimu anowanzo kuwana mhedzisiro iri nani kana ivo vachitsanangura hunhu hwepamberi, chengetedza nzira yekukwira kwevanhu yemakesi emupendero, uye kuteedzera zvese zvakawanikwa zvechigadzirwa uye mutengo wekukanganisa nekufamba kwenguva.
Njodzi & Guardrails
Kugadzirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba.
Infrastructure uye mari yekugadzirisa inowanzotarisirwa pasi.
Chengetedzo uye kucherechedzwa mapundu anogona kukura sezvo masisitimu anowedzera kuoma.
Implementation Roadmap
Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa.
Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.
Benchmark pasi pechokwadi mutoro uye data mamiriro.
Benchmark pasi pechokwadi mutoro uye data mamiriro. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.
Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro.
Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.
Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera.
Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.