Incamake
SwiGLU nigikorwa cyo gufunga amarembo agwiza umurongo umwe ugereranya ibyinjijwe na Swish-ikora ya kabiri ya projection, ikora nk'irembo ryiga, rishingiye ku irembo imbere muri transformateur igaburira imbere. Ihora itezimbere ururimi-rwicyitegererezo, niyo mpamvu hafi ya LLM igezweho ikoresha.
Ibikorwa bya SwiGLU na Gatedi nibikorwa byubaka tekinike bigira ingaruka kumiterere yicyitegererezo, igiciro cyibikorwa remezo, ubukererwe, no kwizerwa mubipimo.
Kwibira cyane
Ihinduramiterere risanzwe rigaburira-imbere ni imirongo ibiri igizwe na ReLU cyangwa GELU hagati. Gates Linear Units, yatanzwe na Dauphin nabandi. muri 2016, gabanya projection ya mbere mo kabiri hanyuma ukoreshe igice kimwe kugirango winjire ikindi ukoresheje ibintu-bigwira. SwiGLU, ikunzwe na Noam Shazeer muri 2020, ikoresha imikorere ya Swish (SiLU) kuri iryo rembo: ibisohoka = (Swish (xW) * (xV)) W2, hamwe na matrica eshatu z'uburemere aho kuba ebyiri. Irembo ryemerera umuyoboro guhitamo cyangwa guhagarika amakuru kurwego. Kuberako kongeramo matrix ya gatatu ikura ibipimo, gushyira mubikorwa bigabanya ibipimo byihishe hafi bibiri bya gatatu kuburyo compte yose iguma igereranijwe na GELU MLP. Ubushakashatsi bwa Shazeer bwerekanye inyungu zishobora kugerwaho, kandi LLaMA, PaLM, na Mistral bose barabyemeje.
Ubushishozi
Swish ni x * sigmoid (beta * x), imikorere yoroshye, itari monotonic, itandukanye na ReLU, yemerera indangagaciro ntoya binyuze. Muri SwiGLU ishami rya 'irembo' Swish (xW) itanga indangagaciro hafi 0 cyangwa 1 zigwiza 'agaciro' ishami xV element-ifite ubwenge, bityo buri gice cyihishe umusanzu uhindurwa nikimenyetso cyize, cyinjiza-gishingiye. Matrix ya gatatu yuburemere nigiciro; bibiri bya gatatu byihishe-ubunini bwamayeri bituma ingengo yimari ya FLOP ihuye na vanilla ibiryo-byimbere.
Kumenya SwiGLU hamwe nibikorwa bya Gate
SwiGLU nigikorwa cyo gufunga amarembo agwiza umurongo umwe ugereranya ibyinjijwe na Swish-ikora ya kabiri ya projection, ikora nk'irembo ryiga, rishingiye ku irembo imbere muri transformateur igaburira imbere. Ihora itezimbere ururimi-rwicyitegererezo, niyo mpamvu hafi ya LLM igezweho ikoresha. Ibikorwa bya SwiGLU na Gatedi nibikorwa byubaka tekinike bigira ingaruka kumiterere yicyitegererezo, igiciro cyibikorwa remezo, ubukererwe, no kwizerwa mubipimo. Kugirango wubake byimbitse, fata ibikorwa bya SwiGLU na Gatedi nkicyitegererezo gikora, ntabwo ari ikintu kimwe: gusobanura ibyagezweho, gusobanura ibitekerezo, no gutandukanya ibyo sisitemu ishobora gukora byizewe nibisaba guca imanza zinzobere.
Mu myitozo, amakipe akomeye akoresha SwiGLU na Gated Activations atezimbere ubwubatsi, amakuru, hamwe nibikorwa remezo birwanya kwizerwa nigiciro. Bandika ibipimo ngenderwaho byerekana intsinzi, bagerageza kurwanya amakuru afatika hamwe nakazi keza, kandi bagasubiramo bashingiye kubikorwa byagaragaye ko batsinzwe aho gutsinda inshuro imwe. Aha niho imyumvire yubumenyi ihinduka mubushobozi burambye kubicuruzwa, politiki, nibikorwa.
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka. Mugihe kimwe, Guhindura igipimo kimwe gishobora guhisha intege nke za sisitemu. Uburyo bukomeye cyane ni uguhuza umuvuduko wikigereranyo hamwe na disipuline yimiyoborere: kuyobora abaderevu, gufata ibimenyetso, gutangaza ibyemezo byicyemezo, no gukomeza kuvugurura uburyo bwo kwirinda nkimyitwarire yicyitegererezo, ibyo abakoresha bategereje, nibisabwa n'amategeko bigenda bihinduka.
Ingaruka z'Ingamba
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Gushyira mu bikorwa Isi
LLaMA, PaLM, na Mistral bisimbuza GELU ibiryo-byimbere hamwe na SwiGLU kugirango bigabanye urujijo kuri compte ingana
Ibipimo byihishe bipimirwa kuri bibiri bya gatatu (8/3 d) kugirango matrike yinyongera yinjirira idatera FLOPs
Uruvange-rwinzobere moderi nka Mixtral ikoresha blike ya SwiGLU nkumuyoboro wa buri muhanga ugaburira
Icyerekezo hamwe na multimodal transformateur baguza GeGLU / SwiGLU amarembo kugirango bateze imbere MLP sublayers
Uburyo bwo Gushyira mu bikorwa
SwiGLU na Gatesi Ibikorwa mubikorwa
LLaMA, PaLM, na Mistral bisimbuza GELU ibiryo-byimbere hamwe na SwiGLU kugirango bigabanye urujijo kuri compte ingana.
LLaMA, PaLM, na Mistral bisimbuza GELU ibiryo-byimbere hamwe na SwiGLU kugirango bigabanye urujijo ku makipe angana kuri compte ubusanzwe babona ibisubizo byiza iyo basobanuye ibipimo byujuje ubuziranenge imbere, bagakomeza inzira yo kuzamura abantu kubibazo bikurikirana, kandi bagakurikirana inyungu zibyara umusaruro nibiciro byamakosa mugihe runaka.
SwiGLU na Gatesi Ibikorwa mubikorwa
Ibipimo byihishe bipimirwa kuri bibiri bya gatatu (8/3 d) kugirango matrike yinyongera yinjirira idatera FLOPs.
Ibipimo byihishe bipimirwa kuri bibiri bya gatatu (8/3 d) kugirango matrike yinyongera yinjirira idatera FLOPs Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo byujuje ubuziranenge imbere, agakomeza inzira yo kuzamura abantu kubibazo byimbere, kandi akurikirana inyungu zibyara umusaruro nibiciro byamakosa mugihe runaka.
SwiGLU na Gatesi Ibikorwa mubikorwa
Imvange-yinzobere moderi nka Mixtral ikoresha blike ya SwiGLU nkumuyoboro ugaburira-umuhanga.
Imvange-yinzobere yerekana urugero nka Mixtral ikoresha blike ya SwiGLU nkumuhanga wimpuguke igaburira imbere-Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo byujuje ubuziranenge imbere, agakomeza inzira yo kuzamura abantu kubibazo, kandi akurikirana inyungu zibyara umusaruro hamwe nibiciro byamakosa mugihe runaka.
SwiGLU na Gatesi Ibikorwa mubikorwa
Icyerekezo hamwe na multimodal transformateur baguza GeGLU / SwiGLU amarembo kugirango batezimbere MLP.
Iyerekwa hamwe na multimodal transformateur baguza GeGLU / SwiGLU kugirango batezimbere MLP sublayers Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo ngenderwaho byimbere, agakomeza inzira yo kuzamura abantu kubibazo, kandi akurikirana inyungu zibyara umusaruro nibiciro byamakosa mugihe runaka.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.