SwiGLU ati Awọn iṣẹ ṣiṣe Gated

An activation function is the nonlinear step inside a neural network that lets it model more than straight lines. In a transformer, the most obvious place it appears is the feed-forward network (FFN), the small two-layer network applied to every token after the attention step. The original 2017 transformer used ReLU there, which keeps positive numbers and zeroes out negatives. A gated activation works differently: it computes two separate linear projections of the same input and multiplies them element by element, so one projection acts as a gate that scales the other up or down for each feature. SwiGLU is the gated variant in which the gate passes through the Swish function. It replaces the first layer and the activation of the FFN, not the attention mechanism.

Step by step, a standard ReLU FFN takes a token vector x of width d, multiplies it by a matrix W1 to get a wider hidden vector, applies ReLU, and multiplies by W2 to return to width d. A SwiGLU FFN uses three matrices. It computes xW and xV, two hidden vectors of the same size. It applies Swish to the first one; Swish(z) equals z times sigmoid(z), a smooth curve that behaves like ReLU for large positive values but lets small negative values through and dips slightly below zero. It multiplies Swish(xW) by xV element by element, and only then projects back with W2. The result is that each hidden feature is the product of a content signal and a learned, input-dependent gate.

The idea has three ancestors. Yann Dauphin and colleagues at Facebook AI Research introduced the gated linear unit (GLU) in December 2016 in "Language Modeling with Gated Convolutional Networks". Their GLU multiplies a linear projection by a sigmoid of a second projection, and they argued that the ungated linear path eases the vanishing-gradient problem in deep networks while keeping nonlinear capacity. Prajit Ramachandran, Barret Zoph and Quoc Le at Google Brain found Swish in October 2017 through an automated search over activation functions, reporting top-1 ImageNet gains of 0.9% for Mobile NASNet-A and 0.6% for Inception-ResNet-v2 when replacing ReLU. Noam Shazeer then combined the two in "GLU Variants Improve Transformer", posted in February 2020, which defined ReGLU, GEGLU and SwiGLU and tested them in the T5 model.

Shazeer's experiment was deliberately fair. Because gated layers have three weight matrices rather than two, he shrank the hidden width by a factor of two thirds, from 3,072 to 2,048 in T5-base, so every variant had the same parameter count and operation count. After 524,288 pre-training steps on the C4 dataset, held-out log-perplexity was 1.677 for ReLU, 1.679 for GELU, 1.633 for GEGLU and 1.636 for SwiGLU. On fine-tuning, the GLUE average rose from 83.80 with ReLU to 84.36 with SwiGLU, and the SuperGLUE average from 72.76 to 74.56. The paper itself offers no theory for why the variants work, and its author attributes their success, tongue in cheek, to divine benevolence.

Adoption followed in the largest models. Google's PaLM paper (April 2022) listed SwiGLU among its architecture choices, citing Shazeer's quality gains over ReLU, GELU and plain Swish. Meta's LLaMA paper (February 2023) adopted SwiGLU, crediting PaLM for the choice, and set the FFN hidden size to two thirds of 4d, where d is the model width. A 2021 study by Sharan Narang and colleagues, "Do Transformer Modifications Transfer Across Implementations and Applications?", is useful context: it found that most proposed transformer changes did not improve results when reimplemented, yet SwiGLU and GeGLU improved pre-training, fine-tuning and supervised training without slowing training.

Compared with the alternatives, the differences are modest but consistent. ReLU is the cheapest and simplest. GELU, which weights each input by the probability that a standard normal variable falls below it, and plain Swish are smooth single-path activations that performed close to ReLU in Shazeer's perplexity table. The gated family, GLU, ReGLU, GEGLU and SwiGLU, all beat the ungated versions in that table, with GEGLU and SwiGLU best. Even the bilinear variant with no nonlinearity at all, just the product of two projections, reached 1.648, better than ReLU, which suggests that the multiplicative gate matters at least as much as the exact curve.

The limitations are practical rather than dramatic. The gains are small in absolute terms, a few hundredths of log-perplexity in the original study, and Shazeer's fine-tuning results are described in the paper as noisy, so single-benchmark comparisons should be read with caution. The extra matrix means three weight reads and three matrix multiplications per FFN, which PaLM acknowledges, and the two intermediate projections both have to be held in memory during training. Choosing the hidden width matters: without the two-thirds reduction a SwiGLU layer has 50% more FFN parameters than its ReLU counterpart, which would make comparisons unfair. Finally, the evidence is empirical; no published theory explains why gating helps.

In practice, SwiGLU costs nothing extra once the width is adjusted. Most deep learning libraries provide Swish under the name SiLU, so a SwiGLU FFN is three linear layers, one SiLU call and one element-wise multiplication. The Swish paper notes the function can be written with a trainable scale beta, but the transformer versions use a fixed beta of 1. Hardware teams sometimes round the two-thirds width to a multiple of a convenient number so the matrices map well onto accelerator tiles. For anyone training a transformer from scratch, the published record supports SwiGLU or GEGLU as a reasonable default.