技術指南

SwiGLU 和門控激活

SwiGLU 是變壓器前饋層中使用的門控激活:輸入被投影兩次,一個副本通過 Swish 函數,兩個副本逐個元素相乘。

聆聽此指南AI 語音敘述 · ~7 min

本頁8 分鐘閱讀
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. 手動追蹤一個 SwiGLU 前饋
  5. 現實世界的實施
  6. SwiGLU 和門控啟動的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

Noam Shazeer 的 2020 年論文表明,它降低了同等計算的複雜性,PaLM 和 LLaMA 採用了它,使其成為大型語言模型設計的標準部分。

重點摘要

  1. SwiGLU 將 Swish(xW) 逐個元素乘以第二個投影 xV,然後用第三個矩陣向下投影。
  2. Noam Shazeer 在 2020 年 2 月的論文 GLU Variants Improve Transformer 中介紹了 SwiGLU 和相關的 GLU 變體。
  3. 為了保持參數相等,門控前饋層將隱藏寬度縮小到通常 4d 的三分之二。
  4. 在基於 T5 的測試中,與匹配計算的 ReLU 相比,SwiGLU 將保留的對數困惑度從 1.677 降低到 1.636。
  5. 2022 年的 PaLM 和 2023 年的 LLaMA 都在前饋層使用 SwiGLU。
  6. 門控線性單元可追溯到 Dauphin 等人 2016 年 Facebook AI Research 的門控卷積語言模型。

深入探討

An activation function is the nonlinear step inside a neural network that lets it model more than straight lines. In a transformer, the most obvious place it appears is the feed-forward network (FFN), the small two-layer network applied to every token after the attention step. The original 2017 transformer used ReLU there, which keeps positive numbers and zeroes out negatives. A gated activation works differently: it computes two separate linear projections of the same input and multiplies them element by element, so one projection acts as a gate that scales the other up or down for each feature. SwiGLU is the gated variant in which the gate passes through the Swish function. It replaces the first layer and the activation of the FFN, not the attention mechanism.

Step by step, a standard ReLU FFN takes a token vector x of width d, multiplies it by a matrix W1 to get a wider hidden vector, applies ReLU, and multiplies by W2 to return to width d. A SwiGLU FFN uses three matrices. It computes xW and xV, two hidden vectors of the same size. It applies Swish to the first one; Swish(z) equals z times sigmoid(z), a smooth curve that behaves like ReLU for large positive values but lets small negative values through and dips slightly below zero. It multiplies Swish(xW) by xV element by element, and only then projects back with W2. The result is that each hidden feature is the product of a content signal and a learned, input-dependent gate.

The idea has three ancestors. Yann Dauphin and colleagues at Facebook AI Research introduced the gated linear unit (GLU) in December 2016 in "Language Modeling with Gated Convolutional Networks". Their GLU multiplies a linear projection by a sigmoid of a second projection, and they argued that the ungated linear path eases the vanishing-gradient problem in deep networks while keeping nonlinear capacity. Prajit Ramachandran, Barret Zoph and Quoc Le at Google Brain found Swish in October 2017 through an automated search over activation functions, reporting top-1 ImageNet gains of 0.9% for Mobile NASNet-A and 0.6% for Inception-ResNet-v2 when replacing ReLU. Noam Shazeer then combined the two in "GLU Variants Improve Transformer", posted in February 2020, which defined ReGLU, GEGLU and SwiGLU and tested them in the T5 model.

Shazeer's experiment was deliberately fair. Because gated layers have three weight matrices rather than two, he shrank the hidden width by a factor of two thirds, from 3,072 to 2,048 in T5-base, so every variant had the same parameter count and operation count. After 524,288 pre-training steps on the C4 dataset, held-out log-perplexity was 1.677 for ReLU, 1.679 for GELU, 1.633 for GEGLU and 1.636 for SwiGLU. On fine-tuning, the GLUE average rose from 83.80 with ReLU to 84.36 with SwiGLU, and the SuperGLUE average from 72.76 to 74.56. The paper itself offers no theory for why the variants work, and its author attributes their success, tongue in cheek, to divine benevolence.

Adoption followed in the largest models. Google's PaLM paper (April 2022) listed SwiGLU among its architecture choices, citing Shazeer's quality gains over ReLU, GELU and plain Swish. Meta's LLaMA paper (February 2023) adopted SwiGLU, crediting PaLM for the choice, and set the FFN hidden size to two thirds of 4d, where d is the model width. A 2021 study by Sharan Narang and colleagues, "Do Transformer Modifications Transfer Across Implementations and Applications?", is useful context: it found that most proposed transformer changes did not improve results when reimplemented, yet SwiGLU and GeGLU improved pre-training, fine-tuning and supervised training without slowing training.

Compared with the alternatives, the differences are modest but consistent. ReLU is the cheapest and simplest. GELU, which weights each input by the probability that a standard normal variable falls below it, and plain Swish are smooth single-path activations that performed close to ReLU in Shazeer's perplexity table. The gated family, GLU, ReGLU, GEGLU and SwiGLU, all beat the ungated versions in that table, with GEGLU and SwiGLU best. Even the bilinear variant with no nonlinearity at all, just the product of two projections, reached 1.648, better than ReLU, which suggests that the multiplicative gate matters at least as much as the exact curve.

The limitations are practical rather than dramatic. The gains are small in absolute terms, a few hundredths of log-perplexity in the original study, and Shazeer's fine-tuning results are described in the paper as noisy, so single-benchmark comparisons should be read with caution. The extra matrix means three weight reads and three matrix multiplications per FFN, which PaLM acknowledges, and the two intermediate projections both have to be held in memory during training. Choosing the hidden width matters: without the two-thirds reduction a SwiGLU layer has 50% more FFN parameters than its ReLU counterpart, which would make comparisons unfair. Finally, the evidence is empirical; no published theory explains why gating helps.

In practice, SwiGLU costs nothing extra once the width is adjusted. Most deep learning libraries provide Swish under the name SiLU, so a SwiGLU FFN is three linear layers, one SiLU call and one element-wise multiplication. The Swish paper notes the function can be written with a trainable scale beta, but the transformer versions use a fixed beta of 1. Hardware teams sometimes round the two-thirds width to a multiple of a convenient number so the matrices map well onto accelerator tiles. For anyone training a transformer from scratch, the published record supports SwiGLU or GEGLU as a reasonable default.

04工作範例

手動追蹤一個 SwiGLU 前饋

  1. 為簡單起見,採用寬度 d = 2 的玩具標記向量 x = [1, -1] 並選擇隱藏寬度 2。

  2. 令 W 為單位矩陣,因此 xW = [1, -1],令 V = [[2, 0], [0, 2]],因此 xV = [2, -2]。

  3. 將 Swish 應用於 xW:Swish(1) = 1 * sigmoid(1),約 0.731,Swish(-1) = -1 * sigmoid(-1),約為 -0.269。

  4. 將每個元素與 xV 相乘:0.731 * 2 約為 1.462,-0.269 * -2 約為 0.538,得到 [1.462, 0.538]。

  5. 用 W2 投影回來;以W2為恆等式,FFN輸出為[1.462,0.538]。

  6. 與使用 W1 = V 的 ReLU FFN 進行比較:ReLU([2, -2]) = [2, 0],因此負特徵完全遺失而不是被門控。

它顯示了什麼

手動計算的 SwiGLU 輸出約為 [1.462, 0.538]。透過在 PyTorch 中透過 torch.nn.function.silu(x @ W) * (x @ V) @ W2 運行相同的數字來檢查它,它應該匹配到小數點後三位,並注意閘如何保持負輸入的縮放版本。

現實世界的實施

PaLM(Google,2022)在每個 MLP 區塊中使用 SwiGLU,指出它需要三個矩陣乘法而不是兩個,但在計算匹配實驗中提高了品質。

LLaMA (Meta, 2023) 以 SwiGLU 取代 ReLU,並將前饋隱藏大小設為 4d 的三分之二而不是 4d,因此參數計數保持接近標準區塊。

在 Shazeer 的基於 T5 的實驗中,將 ReLU 前饋層替換為 SwiGLU,在使用匹配參數的 524,288 個訓練步驟後,將保留的對數困惑度從 1.677 減少到 1.636。

Facebook AI Research(Dauphin 等人,2016)的門控卷積語言模型引入了門控線性單元,並在 WikiText-103 上報告了最先進的結果,並且沒有重複出現。

2021 年 Google 對數十種 Transformer 修改的研究發現,SwiGLU 和 GeGLU 是少數能夠在不減慢訓練速度的情況下改善預訓練和微調的方法之一。

SwiGLU 和門控啟動的未來

The research record on SwiGLU is mostly empirical. Shazeer's 2020 paper offered no explanation for the gains, and the 2021 Narang et al. study warned that many transformer changes fail to transfer across codebases, so independent replications remain the main evidence. Open questions include why multiplicative gating helps so consistently, whether the benefit changes at very large scale, and how gating interacts with other common choices such as RMS normalization, mixture-of-experts layers and low-precision training. On the engineering side, SwiGLU's three matrices and two intermediate activations make it a target for fused GPU kernels and memory-saving recomputation, and accelerator vendors and framework authors continue to publish optimized implementations. Since PaLM in 2022 and LLaMA in 2023, gated feed-forward layers have become a default in openly documented large language model architectures, and new activation proposals are now typically benchmarked against SwiGLU or GEGLU rather than against ReLU.

付諸實踐

為什麼這很重要

  • 成本與預算. 多年來,架構決策決定著效能和營運成本。
  • 更明確的決策. 技術教育幫助團隊選擇正確的堆疊,而不僅僅是最新的堆疊。
  • 品質管控. 更好的工程選擇可以減少生產中的可靠性事故。

留意

  • 優化一項基準測試可以隱藏更廣泛的系統弱點。
  • 基礎設施和維護成本常常被低估。
  • 隨著系統變得更加複雜,安全性和可觀察性差距可能會擴大。

後續步驟

  1. 在實施之前定義延遲、品質和成本目標。
  2. 在實際負載和資料條件下進行基準測試。
  3. 儀器監控錯誤、漂移和使用者影響。
  4. 在擴展之前準備回滾和事件回應路徑。

資料來源與延伸閱讀

  1. arXiv (Google)GLU 變體改進了 Transformer
  2. arXiv (ICML 2017)使用門控卷積網路進行語言建模
  3. arXiv (Google Brain)搜尋激活函數
  4. arXiv (Google)PaLM:透過路徑擴展語言建模
  5. arXiv (Meta AI)LLaMA:開放高效率的基礎語言模型
  6. arXiv (Google Research)Transformer 修改是否可以跨實施和應用程式轉移?

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

參加 SwiGLU 和門控啟動測驗

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是 SwiGLU 和門控啟動?

SwiGLU 是變壓器前饋層中使用的門控激活:輸入被投影兩次,一個副本通過 Swish 函數,兩個副本逐個元素相乘。 Noam Shazeer 的 2020 年論文表明,它降低了同等計算的複雜性,PaLM 和 LLaMA 採用了它,使其成為大型語言模型設計的標準部分。

什麼核心操作定義了門控線性單元?

GLU 依元素將數值投影乘以閘投影,讓網路控制每個維度的資訊流。

SwiGLU 的閘門分支使用哪個激活函數?

SwiGLU 使用 Swish(也稱為 SiLU)作為閘控分支,定義為 x * sigmoid(beta*x)。

SwiGLU 前饋模組使用多少個權重矩陣?

SwiGLU 需要三個矩陣:閘投影、值投影和輸出投影。

為什麼 SwiGLU 層中的隱藏維度通常會縮放至約三分之二?

否則第三個矩陣會增加計算量,因此將隱藏大小縮小到大約 8/3 d 可以保持預算匹配。

誰在 2020 年推廣了 SwiGLU 作為變壓器前饋層?

Noam Shazeer 的 2020 年筆記“GLU Variants Improve Transformer”介紹了 SwiGLU 並展示了其困惑度增益。