Ubuyobozi bwa tekiniki

SwiGLU n'ibikorwa bya Gate

SwiGLU ni umuvuduko w'ibikorwa (gated activation) ukoreshwa mu bice byo gutwara amakuru (feed-forward layers) bya transformers: amakuru yinjira yoherezwa inshuro ebyiri, igice kimwe gipita mu mikorere ya Swish, naho ibice byombi bigakuba buri kintu ku cyarwo (element by element).

Umva iki gitaboAI-byavuzwe · ~7 min

Kuriyi page8 min soma
  1. Incamake
  2. Ibyingenzi byingenzi
  3. Kwibira cyane
  4. Gukurikiza intambwe imwe ya SwiGLU mu gice cyo gutwara amakuru ukoresheje intoki
  5. Gushyira mu bikorwa Isi
  6. Kazoza ka SwiGLU na Gatesi Ibikorwa
  7. Kubishyira mubikorwa
  8. Inkomoko no gusoma
  9. Komeza Ubushakashatsi
  10. Ibibazo bikunze kubazwa

Incamake

Inyandiko ya Noam Shazeer y'umwaka wa 2020 yagaragaje ko yagabanyije perplexity ku bikoresho by'ingufu zingana, kandi PaLM na LLaMA bayakoresheje, bigatuma iba igice cy'ibanze mu miterere y'imodelli nini z'ururimi.

Ibyingenzi byingenzi

  1. SwiGLU ikuba Swish(xW) n'iyoherejwe ya kabiri xV buri kintu ku cyarwo, hanyuma ikayohereza hasi n'ikarita ya gatatu.
  2. Noam Shazeer yavuyemo SwiGLU n'izindi mpinduka za GLU mu nyandiko y'ukwezi kwa Gashyantare 2020 yitwa GLU Variants Improve Transformer.
  3. Kugira ngo umubare w'ibipimo (parameters) uboneke, ibice byo gutwara amakuru bifite umuvuduko w'ibikorwa (gated) bigabanya uburebure bw'imbere (hidden width) kugera ku gice cya kabiri cy'uburebure busanzwe bwa 4d.
  4. Mu bikoresho bya T5-base, SwiGLU yagabanyije log-perplexity y'ibisigaye (held-out) kuva kuri 1.677 ikagera kuri 1.636 ugereranyije na ReLU ku bikoresho by'ingufu zingana.
  5. PaLM mu 2022 na LLaMA mu 2023 byombi bikoresha SwiGLU mu bice byabyo byo gutwara amakuru.
  6. Gated linear unit yavuye mu modelli z'ururimi zifite convolutional zifite umuvuduko w'ibikorwa (gated) za Dauphin na bagenzi be z'umwaka wa 2016 zo kuri Facebook AI Research.

Kwibira cyane

An activation function is the nonlinear step inside a neural network that lets it model more than straight lines. In a transformer, the most obvious place it appears is the feed-forward network (FFN), the small two-layer network applied to every token after the attention step. The original 2017 transformer used ReLU there, which keeps positive numbers and zeroes out negatives. A gated activation works differently: it computes two separate linear projections of the same input and multiplies them element by element, so one projection acts as a gate that scales the other up or down for each feature. SwiGLU is the gated variant in which the gate passes through the Swish function. It replaces the first layer and the activation of the FFN, not the attention mechanism.

Step by step, a standard ReLU FFN takes a token vector x of width d, multiplies it by a matrix W1 to get a wider hidden vector, applies ReLU, and multiplies by W2 to return to width d. A SwiGLU FFN uses three matrices. It computes xW and xV, two hidden vectors of the same size. It applies Swish to the first one; Swish(z) equals z times sigmoid(z), a smooth curve that behaves like ReLU for large positive values but lets small negative values through and dips slightly below zero. It multiplies Swish(xW) by xV element by element, and only then projects back with W2. The result is that each hidden feature is the product of a content signal and a learned, input-dependent gate.

The idea has three ancestors. Yann Dauphin and colleagues at Facebook AI Research introduced the gated linear unit (GLU) in December 2016 in "Language Modeling with Gated Convolutional Networks". Their GLU multiplies a linear projection by a sigmoid of a second projection, and they argued that the ungated linear path eases the vanishing-gradient problem in deep networks while keeping nonlinear capacity. Prajit Ramachandran, Barret Zoph and Quoc Le at Google Brain found Swish in October 2017 through an automated search over activation functions, reporting top-1 ImageNet gains of 0.9% for Mobile NASNet-A and 0.6% for Inception-ResNet-v2 when replacing ReLU. Noam Shazeer then combined the two in "GLU Variants Improve Transformer", posted in February 2020, which defined ReGLU, GEGLU and SwiGLU and tested them in the T5 model.

Shazeer's experiment was deliberately fair. Because gated layers have three weight matrices rather than two, he shrank the hidden width by a factor of two thirds, from 3,072 to 2,048 in T5-base, so every variant had the same parameter count and operation count. After 524,288 pre-training steps on the C4 dataset, held-out log-perplexity was 1.677 for ReLU, 1.679 for GELU, 1.633 for GEGLU and 1.636 for SwiGLU. On fine-tuning, the GLUE average rose from 83.80 with ReLU to 84.36 with SwiGLU, and the SuperGLUE average from 72.76 to 74.56. The paper itself offers no theory for why the variants work, and its author attributes their success, tongue in cheek, to divine benevolence.

Adoption followed in the largest models. Google's PaLM paper (April 2022) listed SwiGLU among its architecture choices, citing Shazeer's quality gains over ReLU, GELU and plain Swish. Meta's LLaMA paper (February 2023) adopted SwiGLU, crediting PaLM for the choice, and set the FFN hidden size to two thirds of 4d, where d is the model width. A 2021 study by Sharan Narang and colleagues, "Do Transformer Modifications Transfer Across Implementations and Applications?", is useful context: it found that most proposed transformer changes did not improve results when reimplemented, yet SwiGLU and GeGLU improved pre-training, fine-tuning and supervised training without slowing training.

Compared with the alternatives, the differences are modest but consistent. ReLU is the cheapest and simplest. GELU, which weights each input by the probability that a standard normal variable falls below it, and plain Swish are smooth single-path activations that performed close to ReLU in Shazeer's perplexity table. The gated family, GLU, ReGLU, GEGLU and SwiGLU, all beat the ungated versions in that table, with GEGLU and SwiGLU best. Even the bilinear variant with no nonlinearity at all, just the product of two projections, reached 1.648, better than ReLU, which suggests that the multiplicative gate matters at least as much as the exact curve.

The limitations are practical rather than dramatic. The gains are small in absolute terms, a few hundredths of log-perplexity in the original study, and Shazeer's fine-tuning results are described in the paper as noisy, so single-benchmark comparisons should be read with caution. The extra matrix means three weight reads and three matrix multiplications per FFN, which PaLM acknowledges, and the two intermediate projections both have to be held in memory during training. Choosing the hidden width matters: without the two-thirds reduction a SwiGLU layer has 50% more FFN parameters than its ReLU counterpart, which would make comparisons unfair. Finally, the evidence is empirical; no published theory explains why gating helps.

In practice, SwiGLU costs nothing extra once the width is adjusted. Most deep learning libraries provide Swish under the name SiLU, so a SwiGLU FFN is three linear layers, one SiLU call and one element-wise multiplication. The Swish paper notes the function can be written with a trainable scale beta, but the transformer versions use a fixed beta of 1. Hardware teams sometimes round the two-thirds width to a multiple of a convenient number so the matrices map well onto accelerator tiles. For anyone training a transformer from scratch, the published record supports SwiGLU or GEGLU as a reasonable default.

04Urugero rwakazi

Gukurikiza intambwe imwe ya SwiGLU mu gice cyo gutwara amakuru ukoresheje intoki

  1. Take a toy token vector x = [1, -1] with width d = 2 and choose a hidden width of 2 for simplicity.

  2. Let W be the identity matrix, so xW = [1, -1], and let V = [[2, 0], [0, 2]], so xV = [2, -2].

  3. Apply Swish to xW: Swish(1) = 1 * sigmoid(1), about 0.731, and Swish(-1) = -1 * sigmoid(-1), about -0.269.

  4. Multiply element by element with xV: 0.731 * 2 is about 1.462 and -0.269 * -2 is about 0.538, giving [1.462, 0.538].

  5. Project back with W2; with W2 as the identity, the FFN output is [1.462, 0.538].

  6. Compare with a ReLU FFN using W1 = V: ReLU([2, -2]) = [2, 0], so the negative feature is lost entirely instead of being gated.

Ibyo byerekana

Igisaruro cya SwiGLU cyabazwe n'intoki cya hafi ya [1.462, 0.538]. Ubigenzure ukoresheje imibare imwe unyuze muri torch.nn.functional.silu(x @ W) * (x @ V) @ W2 muri PyTorch, ikwiye guhura ku myanya itatu nyuma y'icyegeranyo, kandi witegereze uburyo irembo ryazigamye verisiyo yagabanijwe y'ibyinjiye bibi.

Gushyira mu bikorwa Isi

PaLM (Google, 2022) ikoresha SwiGLU muri buri gika cya MLP, ivuga ko ikeneye ugwiza kwa matrix eshatu aho kuba ebyiri ariko ikongera ubuziranenge mu bushakashatsi bw'imibare yagereranijwe.

LLaMA (Meta, 2023) ishyiraho SwiGLU ahabaga ReLU ikaba ishyiraho ubunini bwihishwa bw'ibiryo-imbere kuri bitatu bya kabiri bya 4d aho kuba 4d, bityo umubare w'ibipimo uguma hafi y'igitabo gisanzwe.

Mu bushakashatsi bwa Shazeer bwa T5-base, gusimbuza agace k'ibiryo-imbere ka ReLU na SwiGLU byagabanije log-perplexity yari yaragumishijweho iva kuri 1.677 igera kuri 1.636 nyuma y'intambwe z'imyitozo 524,288 hamwe n'ibipimo byagereranijwe.

Icyitegererezo cy'ururimi rw'amashusho y'intoki cyakozwe na Facebook AI Research (Dauphin et al., 2016) cyazanye urwego rufunze rwa linear (gated linear unit) ndetse kigaragaza ibisubizo byiza kurusha ibindi kuri WikiText-103 nta gusubiramo.

Ubushakashatsi bwa 2021 bwa Google ku mpinduka nyinshi za transformer bwagaragaje ko SwiGLU na GeGLU ziri mu nke zatumye imyitozo y'ibanze n'iyihariye irushaho kuba nziza zitacinze umuvuduko w'imyitozo.

Kazoza ka SwiGLU na Gatesi Ibikorwa

The research record on SwiGLU is mostly empirical. Shazeer's 2020 paper offered no explanation for the gains, and the 2021 Narang et al. study warned that many transformer changes fail to transfer across codebases, so independent replications remain the main evidence. Open questions include why multiplicative gating helps so consistently, whether the benefit changes at very large scale, and how gating interacts with other common choices such as RMS normalization, mixture-of-experts layers and low-precision training. On the engineering side, SwiGLU's three matrices and two intermediate activations make it a target for fused GPU kernels and memory-saving recomputation, and accelerator vendors and framework authors continue to publish optimized implementations. Since PaLM in 2022 and LLaMA in 2023, gated feed-forward layers have become a default in openly documented large language model architectures, and new activation proposals are now typically benchmarked against SwiGLU or GEGLU rather than against ReLU.

Kubishyira mubikorwa

Impamvu ari ngombwa

  • Igiciro na bije. Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
  • Ibyemezo bisobanutse. Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
  • Kugenzura ubuziranenge. Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.

Witondere

  • Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
  • Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
  • Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.

Intambwe ikurikira

  1. Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
  2. Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
  3. Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
  4. Tegura inzira yo gusubiza ibyabaye mbere yo gupima.

Inkomoko no gusoma

  1. arXiv (Google)Ubwoko bwa GLU Buzamura Transformer
  2. arXiv (ICML 2017)Gukora ku Ngero z'Ururimi ukoresheje Inzitizi zifunze za Convolutional
  3. arXiv (Google Brain)Gushakisha Imikorere y'Ibikorwa
  4. arXiv (Google)PaLM: Kuzamura Imitunganyirize y'Ururimi ukoresheje Pathways
  5. arXiv (Meta AI)LLaMA: Ibyitegererezo by'Ururimi by'Ibanze bifunguye kandi bifite Akamaro
  6. arXiv (Google Research)Ese Impinduka za Transformer Zigaragara mu Bikorwa no mu Gukoresha bitandukanye?

Komeza Ubushakashatsi

Ibaruwa y'amakuru y'ubuntu

Ibuka amakuru ya buri munsi ya AI

Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.

Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.

Isuzume

Kora ikizamini cya SwiGLU n'Ibyagurisha Bifunze

Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.

Tangira ikibazo

Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano

Ibibazo bikunze kubazwa

Niki SwiGLU na Gatesi Ibikorwa?

SwiGLU ni igikorwa gifunze gikoreshwa imbere mu nzego z'ibiryo-imbere za transformers: ibyinjiye bipimishwa kabiri, kopi imwe inyura mu gikorwa cya Swish, maze kopi zombi zigwizwa n'ikintu ku kindi. Impapuro za Noam Shazeer z'umwaka wa 2020 zagaragaje ko zagabanije ubwinshi (perplexity) ku mibare ingana, kandi PaLM na LLaMA babyemeye, bigatuma biba igice gisanzwe mu gushushanya ibyitegererezo binini by'indimi.

Ni ikihe gikorwa cy'ibanze gisobanura umurongo wa Gateur?

GLU igwiza agaciro projection kumarembo yerekana ibintu-neza, kureka imiyoboro igenzura amakuru atemba murwego.

Nibihe bikorwa byo gukora SwiGLU ikoresha ishami ryayo?

SwiGLU ikoresha Swish, nanone yitwa SiLU, isobanurwa nka x * sigmoid (beta * x), ku ishami ryinjira.

Ni bangahe matrices yuburemere SwiGLU igaburira-imbere ikumira?

SwiGLU ikeneye matrices eshatu: umushinga w irembo, agaciro kerekana, hamwe nibisohoka.

Ni ukubera iki igipimo cyihishe gisanzwe gipima hafi bibiri bya gatatu mubice bya SwiGLU?

Matrix ya gatatu ubundi yazamura compte, bityo kugabanya ubunini bwihishe hafi 8/3 d bituma ingengo yimari ihuye.

Ninde wamamaje SwiGLU ya transfert yo kugaburira imbere muri 2020?

Noam Shazeer inoti ya 2020 'GLU Variants Impinduka Transformer' yerekanye SwiGLU kandi yerekana inyungu zayo zitangaje.