Reseaux de squeeze-et-excitation

A convolutional neural network (CNN) builds up an image representation in layers. Each layer produces a stack of feature maps, one per channel, where each channel responds to some learned pattern such as an edge, a texture or, deeper in the network, part of an object. A standard convolution mixes information across channels only within a small neighborhood of pixels. A squeeze-and-excitation block adds a cheap step that looks at every channel across the whole image and decides how much each channel should matter for this particular input. It then multiplies each channel by that weight. The authors call this feature recalibration, and it is a form of attention applied to channels rather than to positions or tokens.

The block has three steps. Squeeze: global average pooling collapses each channel's height by width map into a single number, the channel's average activation, producing a vector with one value per channel. Excitation: that vector passes through a small two-layer network. The first fully connected layer shrinks it by a reduction ratio r, 16 by default, followed by a ReLU; the second expands it back to the original number of channels, followed by a sigmoid that maps each value to between 0 and 1. Scale: each channel of the original feature map is multiplied by its weight. The sigmoid was chosen deliberately so that several channels can be emphasized at once, rather than forcing a single winner as a softmax would.

The block is meant to be dropped into existing architectures. In a residual network such as ResNet, it is applied to the output of the residual branch just before that branch is added back to the identity shortcut. The same pattern works for ResNeXt and Inception-style modules. Because the block only needs a feature map as input, it can be placed after almost any convolutional stage, which is a large part of why it spread so quickly.

The work came from Jie Hu, Li Shen and Gang Sun, with Samuel Albanie and Enhua Wu on the extended paper, affiliated with Momenta and the University of Oxford according to the official repository, which implements the networks in the Caffe framework. The paper was first posted to arXiv in September 2017 and published at CVPR 2018. SENets formed the basis of the team's winning entry in the ILSVRC 2017 classification competition: a small ensemble reached a 2.251% top-5 error on the test set, roughly a 25% relative improvement over the 2016 winner's 2.991%. A single SENet-154 model, built on a modified ResNeXt, reached 18.68% top-1 and 4.47% top-5 error with a 224 by 224 center crop.

The gains were consistent across networks. SE-ResNet-50 reached a 6.62% top-5 validation error, against 7.48% for ResNet-50 and 6.52% for ResNet-101, which needs 7.58 GFLOPs. SE-ResNet-101 at 6.07% beat ResNet-152 at 6.34%. SE-ResNeXt-50 at 5.49% beat ResNeXt-101 at 5.57%, and an SE version of Inception-ResNet-v2 improved from 5.21% to 4.79%. Outside classification, SE-ResNet-50 improved COCO object detection average precision by 2.4 points and SE-ResNet-152 lowered Places365 scene-classification top-5 error from 11.61% to 11.01%. The authors also observed that early SE blocks behave in a class-agnostic way, strengthening shared low-level features, while later blocks become increasingly class-specific.

The idea spread into efficient mobile models and inspired many variants. According to the MobileNetV3 paper, MnasNet added lightweight SE-based attention to the MobileNetV2 bottleneck, and MobileNetV3 kept it with a fixed bottleneck size and a hard sigmoid. EfficientNet-B0, published in 2019, uses the same SE-augmented mobile block. CBAM (Convolutional Block Attention Module, 2018) extended the idea by computing channel attention followed by spatial attention, a map over image positions. ECA-Net (2019) argued that the dimensionality reduction in SE's excitation step hurts, and replaced the two fully connected layers with a single 1D convolution across neighboring channels, adding only 80 parameters to ResNet-50 against SE's roughly 2.5 million, while reporting a 2.28-point top-1 gain over the plain ResNet-50.

Compared with alternatives, SE's appeal is its cost-to-benefit ratio. Making a network deeper, such as going from ResNet-50 to ResNet-101, roughly doubles compute; adding SE blocks to ResNet-50 adds 0.26% more FLOPs for a similar accuracy gain. Spatial attention methods such as CBAM's second stage, or the trunk-and-mask attention of earlier residual attention networks, can capture where to look but cost more. Self-attention as used in vision transformers models relationships between all positions, which SE does not attempt; SE only models relationships between channels, using one global summary per channel.

The limitations are worth knowing. The parameter cost is not negligible: SE adds about 2.5 million parameters to ResNet-50's 25 million, a 10% increase, most of it in the last stage, which the authors show can be removed with less than 0.1% top-5 error cost, cutting the increase to 4%. Global average pooling throws away all spatial information, so SE cannot tell where in the image a feature fired. The excitation design is sensitive: replacing the sigmoid with ReLU made SE-ResNet-50 worse than the baseline ResNet-50. FLOPs understate real cost on some hardware, because the pooling and channel-wise multiplication add memory traffic; the official repository notes that the authors re-implemented global average pooling on GPU because the stock version was inefficient, and MobileNetV3 replaced the sigmoid partly for efficiency on phones.