技術指南

擠壓和激勵網絡

A squeeze-and-excitation (SE) block is a small add-on for convolutional neural networks that summarizes each feature channel across the whole image, then learns a weight between 0 and 1 for every channel and rescales the channels accordingly.

聆聽此指南AI 語音敘述 · ~8 min

本頁閱讀時間 9 分鐘
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. Computing an SE block by hand on a tiny feature map
  5. 現實世界的實施
  6. 擠壓和激勵網路的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

Introduced by Jie Hu and colleagues in 2017, SE networks won the ILSVRC 2017 image classification challenge and became a standard component of efficient vision models.

重點摘要

  1. An SE block averages each channel over the image, passes the result through a small bottleneck network with a sigmoid, and rescales channels.
  2. SENets won the ILSVRC 2017 classification challenge with a 2.251% top-5 error on the ImageNet test set.
  3. SE-ResNet-50 improved top-5 error from 7.48% to 6.62% for only 0.26% more FLOPs than ResNet-50.
  4. SE blocks add about 10% more parameters to ResNet-50, mostly in the final stage with the most channels.
  5. EfficientNet and MobileNetV3 both build squeeze-and-excitation into their main mobile convolution blocks.
  6. Replacing the SE sigmoid with ReLU made SE-ResNet-50 perform worse than the original ResNet-50 in the paper's ablation.

深入探討

A convolutional neural network (CNN) builds up an image representation in layers. Each layer produces a stack of feature maps, one per channel, where each channel responds to some learned pattern such as an edge, a texture or, deeper in the network, part of an object. A standard convolution mixes information across channels only within a small neighborhood of pixels. A squeeze-and-excitation block adds a cheap step that looks at every channel across the whole image and decides how much each channel should matter for this particular input. It then multiplies each channel by that weight. The authors call this feature recalibration, and it is a form of attention applied to channels rather than to positions or tokens.

The block has three steps. Squeeze: global average pooling collapses each channel's height by width map into a single number, the channel's average activation, producing a vector with one value per channel. Excitation: that vector passes through a small two-layer network. The first fully connected layer shrinks it by a reduction ratio r, 16 by default, followed by a ReLU; the second expands it back to the original number of channels, followed by a sigmoid that maps each value to between 0 and 1. Scale: each channel of the original feature map is multiplied by its weight. The sigmoid was chosen deliberately so that several channels can be emphasized at once, rather than forcing a single winner as a softmax would.

The block is meant to be dropped into existing architectures. In a residual network such as ResNet, it is applied to the output of the residual branch just before that branch is added back to the identity shortcut. The same pattern works for ResNeXt and Inception-style modules. Because the block only needs a feature map as input, it can be placed after almost any convolutional stage, which is a large part of why it spread so quickly.

The work came from Jie Hu, Li Shen and Gang Sun, with Samuel Albanie and Enhua Wu on the extended paper, affiliated with Momenta and the University of Oxford according to the official repository, which implements the networks in the Caffe framework. The paper was first posted to arXiv in September 2017 and published at CVPR 2018. SENets formed the basis of the team's winning entry in the ILSVRC 2017 classification competition: a small ensemble reached a 2.251% top-5 error on the test set, roughly a 25% relative improvement over the 2016 winner's 2.991%. A single SENet-154 model, built on a modified ResNeXt, reached 18.68% top-1 and 4.47% top-5 error with a 224 by 224 center crop.

The gains were consistent across networks. SE-ResNet-50 reached a 6.62% top-5 validation error, against 7.48% for ResNet-50 and 6.52% for ResNet-101, which needs 7.58 GFLOPs. SE-ResNet-101 at 6.07% beat ResNet-152 at 6.34%. SE-ResNeXt-50 at 5.49% beat ResNeXt-101 at 5.57%, and an SE version of Inception-ResNet-v2 improved from 5.21% to 4.79%. Outside classification, SE-ResNet-50 improved COCO object detection average precision by 2.4 points and SE-ResNet-152 lowered Places365 scene-classification top-5 error from 11.61% to 11.01%. The authors also observed that early SE blocks behave in a class-agnostic way, strengthening shared low-level features, while later blocks become increasingly class-specific.

The idea spread into efficient mobile models and inspired many variants. According to the MobileNetV3 paper, MnasNet added lightweight SE-based attention to the MobileNetV2 bottleneck, and MobileNetV3 kept it with a fixed bottleneck size and a hard sigmoid. EfficientNet-B0, published in 2019, uses the same SE-augmented mobile block. CBAM (Convolutional Block Attention Module, 2018) extended the idea by computing channel attention followed by spatial attention, a map over image positions. ECA-Net (2019) argued that the dimensionality reduction in SE's excitation step hurts, and replaced the two fully connected layers with a single 1D convolution across neighboring channels, adding only 80 parameters to ResNet-50 against SE's roughly 2.5 million, while reporting a 2.28-point top-1 gain over the plain ResNet-50.

Compared with alternatives, SE's appeal is its cost-to-benefit ratio. Making a network deeper, such as going from ResNet-50 to ResNet-101, roughly doubles compute; adding SE blocks to ResNet-50 adds 0.26% more FLOPs for a similar accuracy gain. Spatial attention methods such as CBAM's second stage, or the trunk-and-mask attention of earlier residual attention networks, can capture where to look but cost more. Self-attention as used in vision transformers models relationships between all positions, which SE does not attempt; SE only models relationships between channels, using one global summary per channel.

The limitations are worth knowing. The parameter cost is not negligible: SE adds about 2.5 million parameters to ResNet-50's 25 million, a 10% increase, most of it in the last stage, which the authors show can be removed with less than 0.1% top-5 error cost, cutting the increase to 4%. Global average pooling throws away all spatial information, so SE cannot tell where in the image a feature fired. The excitation design is sensitive: replacing the sigmoid with ReLU made SE-ResNet-50 worse than the baseline ResNet-50. FLOPs understate real cost on some hardware, because the pooling and channel-wise multiplication add memory traffic; the official repository notes that the authors re-implemented global average pooling on GPU because the stock version was inefficient, and MobileNetV3 replaced the sigmoid partly for efficiency on phones.

04工作範例

Computing an SE block by hand on a tiny feature map

  1. Take a feature map with 4 channels, each 2 by 2 pixels, where channel averages work out to z = [2.0, 0.5, 1.0, 0.0].

  2. Choose a reduction ratio r = 2, so the first fully connected layer maps 4 values to 2 and the second maps 2 back to 4.

  3. Apply the first layer and a ReLU; with example weights the hidden vector might be [1.5, 0.0], since any negative result is clipped to zero.

  4. Apply the second layer and a sigmoid; example pre-activations of [2.0, -1.0, 0.5, -2.0] become weights of about [0.88, 0.27, 0.62, 0.12].

  5. Multiply every pixel in channel 1 by 0.88, channel 2 by 0.27, channel 3 by 0.62 and channel 4 by 0.12.

  6. Count the added parameters: 4 times 2 plus 2 times 4 equals 16 weights, matching the formula 2C squared divided by r.

它顯示了什麼

A recalibrated feature map in which channel 1 is nearly unchanged and channel 4 is strongly suppressed for this input. Check it by implementing the same block in PyTorch with nn.AdaptiveAvgPool2d, two nn.Linear layers, ReLU and sigmoid, loading the same weights, and confirming the outputs match.

現實世界的實施

The SENet ensemble that won the ILSVRC 2017 classification task reached a 2.251% top-5 error on the ImageNet test set, about 25% lower than the 2016 winner's 2.991%.

Adding SE blocks to ResNet-50 cut ImageNet top-5 validation error from 7.48% to 6.62% while raising compute from 3.86 to 3.87 GFLOPs, close to the much deeper ResNet-101.

Google's EfficientNet family uses the mobile inverted bottleneck block with squeeze-and-excitation added as the main building block of its baseline EfficientNet-B0 network.

MobileNetV3 places SE modules after the depthwise convolution, fixes their bottleneck at one quarter of the expansion channels, and swaps the sigmoid for a cheaper hard sigmoid for phones.

Using SE-ResNet-50 instead of ResNet-50 as the backbone of a Faster R-CNN detector raised COCO average precision by 2.4 points, a 6.3% relative improvement, in the SE paper's tests.

擠壓和激勵網路的未來

Channel attention has become a routine design option rather than an active frontier, and most research since 2018 has refined the SE recipe. CBAM (2018) added a spatial attention stage, ECA-Net (2019) removed the dimensionality reduction to cut parameters to a handful, and mobile architectures such as MobileNetV3 and EfficientNet (both 2019) tuned the block's placement, width and activation for real hardware latency. Open questions remain practical. The SE authors suggested that a single reduction ratio throughout a network may not be optimal, and that more sophisticated squeeze operators than average pooling could help. Measured speed on specific devices often differs from FLOP counts, so architecture searches increasingly optimize latency directly, as the MobileNetV3 paper did. As vision transformers and hybrid convolution-attention models spread, the role of a separate channel-attention block in those designs is still being evaluated, and published ablations on standard benchmarks such as ImageNet and COCO remain the reliable way to judge whether SE-style blocks still pay for their cost in a given architecture.

付諸實踐

為什麼這很重要

  • 成本與預算. 多年來,架構決策決定著效能和營運成本。
  • 更明確的決策. 技術教育幫助團隊選擇正確的堆疊,而不僅僅是最新的堆疊。
  • 品質管控. 更好的工程選擇可以減少生產中的可靠性事故。

留意

  • 優化一項基準測試可以隱藏更廣泛的系統弱點。
  • 基礎設施和維護成本常常被低估。
  • 隨著系統變得更加複雜,安全性和可觀察性差距可能會擴大。

後續步驟

  1. 在實施之前定義延遲、品質和成本目標。
  2. 在實際負載和資料條件下進行基準測試。
  3. 儀器監控錯誤、漂移和使用者影響。
  4. 在擴展之前準備回滾和事件回應路徑。

資料來源與延伸閱讀

  1. arXiv (CVPR 2018)擠壓和激勵網絡
  2. GitHub (hujie-frank)SENet: Squeeze-and-Excitation Networks (official Caffe implementation)
  3. arXiv (Google)Searching for MobileNetV3
  4. arXiv (Google)EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
  5. arXiv (KAIST, Lunit, Adobe Research)CBAM: Convolutional Block Attention Module
  6. arXivECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

Take the Squeeze-and-Excitation Networks quiz

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是擠壓和激勵網絡?

A squeeze-and-excitation (SE) block is a small add-on for convolutional neural networks that summarizes each feature channel across the whole image, then learns a weight between 0 and 1 for every channel and rescales the channels accordingly. Introduced by Jie Hu and colleagues in 2017, SE networks won the ILSVRC 2017 image classification challenge and became a standard component of efficient vision models.

SE 區塊中的「擠壓」步驟有什麼作用?

擠壓應用全域平均池化,將每個特徵圖折疊成每個通道的單一描述符。

「激勵」步驟輸出什麼?

激勵將壓縮向量傳遞通過兩個以 s 形曲線結尾的 FC 層,產生每個通道的縮放權重。

激勵權重如何應用於原始特徵圖?

每個通道的特徵圖乘以其學習的權重,放大或抑制該通道。

為什麼激勵使用具有減速比(通常為16)的瓶頸?

瓶頸縮小然後恢復通道尺寸,因此 SE 區塊的成本僅增加幾個百分點。

SENet 在 2017 年贏得了哪些比賽?

SENet 贏得了 ILSVRC 2017 影像分類挑戰賽,將 top-5 錯誤率推至約 2.25%。