HejuruUbuyobozi bukurikira
Igikorwa cyo Kwishura Ibicuruzwa
Tekiniki
Ubuyobozi bwa tekiniki
SwiGLU ni umuvuduko w'ibikorwa (gated activation) ukoreshwa mu bice byo gutwara amakuru (feed-forward layers) bya transformers: amakuru yinjira yoherezwa inshuro ebyiri, igice kimwe gipita mu mikorere ya Swish, naho ibice byombi bigakuba buri kintu ku cyarwo (element by element).
Umva iki gitaboAI-byavuzwe · ~7 min
Inyandiko ya Noam Shazeer y'umwaka wa 2020 yagaragaje ko yagabanyije perplexity ku bikoresho by'ingufu zingana, kandi PaLM na LLaMA bayakoresheje, bigatuma iba igice cy'ibanze mu miterere y'imodelli nini z'ururimi.
An activation function is the nonlinear step inside a neural network that lets it model more than straight lines. In a transformer, the most obvious place it appears is the feed-forward network (FFN), the small two-layer network applied to every token after the attention step. The original 2017 transformer used ReLU there, which keeps positive numbers and zeroes out negatives. A gated activation works differently: it computes two separate linear projections of the same input and multiplies them element by element, so one projection acts as a gate that scales the other up or down for each feature. SwiGLU is the gated variant in which the gate passes through the Swish function. It replaces the first layer and the activation of the FFN, not the attention mechanism.
Step by step, a standard ReLU FFN takes a token vector x of width d, multiplies it by a matrix W1 to get a wider hidden vector, applies ReLU, and multiplies by W2 to return to width d. A SwiGLU FFN uses three matrices. It computes xW and xV, two hidden vectors of the same size. It applies Swish to the first one; Swish(z) equals z times sigmoid(z), a smooth curve that behaves like ReLU for large positive values but lets small negative values through and dips slightly below zero. It multiplies Swish(xW) by xV element by element, and only then projects back with W2. The result is that each hidden feature is the product of a content signal and a learned, input-dependent gate.
The idea has three ancestors. Yann Dauphin and colleagues at Facebook AI Research introduced the gated linear unit (GLU) in December 2016 in "Language Modeling with Gated Convolutional Networks". Their GLU multiplies a linear projection by a sigmoid of a second projection, and they argued that the ungated linear path eases the vanishing-gradient problem in deep networks while keeping nonlinear capacity. Prajit Ramachandran, Barret Zoph and Quoc Le at Google Brain found Swish in October 2017 through an automated search over activation functions, reporting top-1 ImageNet gains of 0.9% for Mobile NASNet-A and 0.6% for Inception-ResNet-v2 when replacing ReLU. Noam Shazeer then combined the two in "GLU Variants Improve Transformer", posted in February 2020, which defined ReGLU, GEGLU and SwiGLU and tested them in the T5 model.
Shazeer's experiment was deliberately fair. Because gated layers have three weight matrices rather than two, he shrank the hidden width by a factor of two thirds, from 3,072 to 2,048 in T5-base, so every variant had the same parameter count and operation count. After 524,288 pre-training steps on the C4 dataset, held-out log-perplexity was 1.677 for ReLU, 1.679 for GELU, 1.633 for GEGLU and 1.636 for SwiGLU. On fine-tuning, the GLUE average rose from 83.80 with ReLU to 84.36 with SwiGLU, and the SuperGLUE average from 72.76 to 74.56. The paper itself offers no theory for why the variants work, and its author attributes their success, tongue in cheek, to divine benevolence.
Adoption followed in the largest models. Google's PaLM paper (April 2022) listed SwiGLU among its architecture choices, citing Shazeer's quality gains over ReLU, GELU and plain Swish. Meta's LLaMA paper (February 2023) adopted SwiGLU, crediting PaLM for the choice, and set the FFN hidden size to two thirds of 4d, where d is the model width. A 2021 study by Sharan Narang and colleagues, "Do Transformer Modifications Transfer Across Implementations and Applications?", is useful context: it found that most proposed transformer changes did not improve results when reimplemented, yet SwiGLU and GeGLU improved pre-training, fine-tuning and supervised training without slowing training.
Compared with the alternatives, the differences are modest but consistent. ReLU is the cheapest and simplest. GELU, which weights each input by the probability that a standard normal variable falls below it, and plain Swish are smooth single-path activations that performed close to ReLU in Shazeer's perplexity table. The gated family, GLU, ReGLU, GEGLU and SwiGLU, all beat the ungated versions in that table, with GEGLU and SwiGLU best. Even the bilinear variant with no nonlinearity at all, just the product of two projections, reached 1.648, better than ReLU, which suggests that the multiplicative gate matters at least as much as the exact curve.
The limitations are practical rather than dramatic. The gains are small in absolute terms, a few hundredths of log-perplexity in the original study, and Shazeer's fine-tuning results are described in the paper as noisy, so single-benchmark comparisons should be read with caution. The extra matrix means three weight reads and three matrix multiplications per FFN, which PaLM acknowledges, and the two intermediate projections both have to be held in memory during training. Choosing the hidden width matters: without the two-thirds reduction a SwiGLU layer has 50% more FFN parameters than its ReLU counterpart, which would make comparisons unfair. Finally, the evidence is empirical; no published theory explains why gating helps.
In practice, SwiGLU costs nothing extra once the width is adjusted. Most deep learning libraries provide Swish under the name SiLU, so a SwiGLU FFN is three linear layers, one SiLU call and one element-wise multiplication. The Swish paper notes the function can be written with a trainable scale beta, but the transformer versions use a fixed beta of 1. Hardware teams sometimes round the two-thirds width to a multiple of a convenient number so the matrices map well onto accelerator tiles. For anyone training a transformer from scratch, the published record supports SwiGLU or GEGLU as a reasonable default.
04Urugero rwakazi
Take a toy token vector x = [1, -1] with width d = 2 and choose a hidden width of 2 for simplicity.
Let W be the identity matrix, so xW = [1, -1], and let V = [[2, 0], [0, 2]], so xV = [2, -2].
Apply Swish to xW: Swish(1) = 1 * sigmoid(1), about 0.731, and Swish(-1) = -1 * sigmoid(-1), about -0.269.
Multiply element by element with xV: 0.731 * 2 is about 1.462 and -0.269 * -2 is about 0.538, giving [1.462, 0.538].
Project back with W2; with W2 as the identity, the FFN output is [1.462, 0.538].
Compare with a ReLU FFN using W1 = V: ReLU([2, -2]) = [2, 0], so the negative feature is lost entirely instead of being gated.
Ibyo byerekana
Igisaruro cya SwiGLU cyabazwe n'intoki cya hafi ya [1.462, 0.538]. Ubigenzure ukoresheje imibare imwe unyuze muri torch.nn.functional.silu(x @ W) * (x @ V) @ W2 muri PyTorch, ikwiye guhura ku myanya itatu nyuma y'icyegeranyo, kandi witegereze uburyo irembo ryazigamye verisiyo yagabanijwe y'ibyinjiye bibi.
PaLM (Google, 2022) ikoresha SwiGLU muri buri gika cya MLP, ivuga ko ikeneye ugwiza kwa matrix eshatu aho kuba ebyiri ariko ikongera ubuziranenge mu bushakashatsi bw'imibare yagereranijwe.
LLaMA (Meta, 2023) ishyiraho SwiGLU ahabaga ReLU ikaba ishyiraho ubunini bwihishwa bw'ibiryo-imbere kuri bitatu bya kabiri bya 4d aho kuba 4d, bityo umubare w'ibipimo uguma hafi y'igitabo gisanzwe.
Mu bushakashatsi bwa Shazeer bwa T5-base, gusimbuza agace k'ibiryo-imbere ka ReLU na SwiGLU byagabanije log-perplexity yari yaragumishijweho iva kuri 1.677 igera kuri 1.636 nyuma y'intambwe z'imyitozo 524,288 hamwe n'ibipimo byagereranijwe.
Icyitegererezo cy'ururimi rw'amashusho y'intoki cyakozwe na Facebook AI Research (Dauphin et al., 2016) cyazanye urwego rufunze rwa linear (gated linear unit) ndetse kigaragaza ibisubizo byiza kurusha ibindi kuri WikiText-103 nta gusubiramo.
Ubushakashatsi bwa 2021 bwa Google ku mpinduka nyinshi za transformer bwagaragaje ko SwiGLU na GeGLU ziri mu nke zatumye imyitozo y'ibanze n'iyihariye irushaho kuba nziza zitacinze umuvuduko w'imyitozo.
The research record on SwiGLU is mostly empirical. Shazeer's 2020 paper offered no explanation for the gains, and the 2021 Narang et al. study warned that many transformer changes fail to transfer across codebases, so independent replications remain the main evidence. Open questions include why multiplicative gating helps so consistently, whether the benefit changes at very large scale, and how gating interacts with other common choices such as RMS normalization, mixture-of-experts layers and low-precision training. On the engineering side, SwiGLU's three matrices and two intermediate activations make it a target for fused GPU kernels and memory-saving recomputation, and accelerator vendors and framework authors continue to publish optimized implementations. Since PaLM in 2022 and LLaMA in 2023, gated feed-forward layers have become a default in openly documented large language model architectures, and new activation proposals are now typically benchmarked against SwiGLU or GEGLU rather than against ReLU.
Ibaruwa y'amakuru y'ubuntu
Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.
Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.
Isuzume
Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.
Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano
SwiGLU ni igikorwa gifunze gikoreshwa imbere mu nzego z'ibiryo-imbere za transformers: ibyinjiye bipimishwa kabiri, kopi imwe inyura mu gikorwa cya Swish, maze kopi zombi zigwizwa n'ikintu ku kindi. Impapuro za Noam Shazeer z'umwaka wa 2020 zagaragaje ko zagabanije ubwinshi (perplexity) ku mibare ingana, kandi PaLM na LLaMA babyemeye, bigatuma biba igice gisanzwe mu gushushanya ibyitegererezo binini by'indimi.
GLU igwiza agaciro projection kumarembo yerekana ibintu-neza, kureka imiyoboro igenzura amakuru atemba murwego.
SwiGLU ikoresha Swish, nanone yitwa SiLU, isobanurwa nka x * sigmoid (beta * x), ku ishami ryinjira.
SwiGLU ikeneye matrices eshatu: umushinga w irembo, agaciro kerekana, hamwe nibisohoka.
Matrix ya gatatu ubundi yazamura compte, bityo kugabanya ubunini bwihishe hafi 8/3 d bituma ingengo yimari ihuye.
Noam Shazeer inoti ya 2020 'GLU Variants Impinduka Transformer' yerekanye SwiGLU kandi yerekana inyungu zayo zitangaje.
Komeza wige
Abandi bayobozi batoranijwe kuriyi ngingo
HejuruUbuyobozi bukurikira
Igikorwa cyo Kwishura Ibicuruzwa
Tekiniki