GUIDA TECNICA

Reti di compressione ed eccitazione

A squeeze-and-excitation (SE) block is a small add-on for convolutional neural networks that summarizes each feature channel across the whole image, then learns a weight between 0 and 1 for every channel and rescales the channels accordingly.

Ascolta questa guidaNarrato dall'IA · ~8 min

In questa pagina9 minuti di lettura
  1. Panoramica
  2. Punti chiave
  3. Immersione profonda
  4. Computing an SE block by hand on a tiny feature map
  5. Implementazione nel mondo reale
  6. Il futuro delle reti di compressione ed eccitazione
  7. Metterlo in pratica
  8. Fonti e approfondimenti
  9. Continua a esplorare
  10. Domande frequenti

Panoramica

Introduced by Jie Hu and colleagues in 2017, SE networks won the ILSVRC 2017 image classification challenge and became a standard component of efficient vision models.

Punti chiave

  1. An SE block averages each channel over the image, passes the result through a small bottleneck network with a sigmoid, and rescales channels.
  2. SENets won the ILSVRC 2017 classification challenge with a 2.251% top-5 error on the ImageNet test set.
  3. SE-ResNet-50 improved top-5 error from 7.48% to 6.62% for only 0.26% more FLOPs than ResNet-50.
  4. SE blocks add about 10% more parameters to ResNet-50, mostly in the final stage with the most channels.
  5. EfficientNet and MobileNetV3 both build squeeze-and-excitation into their main mobile convolution blocks.
  6. Replacing the SE sigmoid with ReLU made SE-ResNet-50 perform worse than the original ResNet-50 in the paper's ablation.

Immersione profonda

A convolutional neural network (CNN) builds up an image representation in layers. Each layer produces a stack of feature maps, one per channel, where each channel responds to some learned pattern such as an edge, a texture or, deeper in the network, part of an object. A standard convolution mixes information across channels only within a small neighborhood of pixels. A squeeze-and-excitation block adds a cheap step that looks at every channel across the whole image and decides how much each channel should matter for this particular input. It then multiplies each channel by that weight. The authors call this feature recalibration, and it is a form of attention applied to channels rather than to positions or tokens.

The block has three steps. Squeeze: global average pooling collapses each channel's height by width map into a single number, the channel's average activation, producing a vector with one value per channel. Excitation: that vector passes through a small two-layer network. The first fully connected layer shrinks it by a reduction ratio r, 16 by default, followed by a ReLU; the second expands it back to the original number of channels, followed by a sigmoid that maps each value to between 0 and 1. Scale: each channel of the original feature map is multiplied by its weight. The sigmoid was chosen deliberately so that several channels can be emphasized at once, rather than forcing a single winner as a softmax would.

The block is meant to be dropped into existing architectures. In a residual network such as ResNet, it is applied to the output of the residual branch just before that branch is added back to the identity shortcut. The same pattern works for ResNeXt and Inception-style modules. Because the block only needs a feature map as input, it can be placed after almost any convolutional stage, which is a large part of why it spread so quickly.

The work came from Jie Hu, Li Shen and Gang Sun, with Samuel Albanie and Enhua Wu on the extended paper, affiliated with Momenta and the University of Oxford according to the official repository, which implements the networks in the Caffe framework. The paper was first posted to arXiv in September 2017 and published at CVPR 2018. SENets formed the basis of the team's winning entry in the ILSVRC 2017 classification competition: a small ensemble reached a 2.251% top-5 error on the test set, roughly a 25% relative improvement over the 2016 winner's 2.991%. A single SENet-154 model, built on a modified ResNeXt, reached 18.68% top-1 and 4.47% top-5 error with a 224 by 224 center crop.

The gains were consistent across networks. SE-ResNet-50 reached a 6.62% top-5 validation error, against 7.48% for ResNet-50 and 6.52% for ResNet-101, which needs 7.58 GFLOPs. SE-ResNet-101 at 6.07% beat ResNet-152 at 6.34%. SE-ResNeXt-50 at 5.49% beat ResNeXt-101 at 5.57%, and an SE version of Inception-ResNet-v2 improved from 5.21% to 4.79%. Outside classification, SE-ResNet-50 improved COCO object detection average precision by 2.4 points and SE-ResNet-152 lowered Places365 scene-classification top-5 error from 11.61% to 11.01%. The authors also observed that early SE blocks behave in a class-agnostic way, strengthening shared low-level features, while later blocks become increasingly class-specific.

The idea spread into efficient mobile models and inspired many variants. According to the MobileNetV3 paper, MnasNet added lightweight SE-based attention to the MobileNetV2 bottleneck, and MobileNetV3 kept it with a fixed bottleneck size and a hard sigmoid. EfficientNet-B0, published in 2019, uses the same SE-augmented mobile block. CBAM (Convolutional Block Attention Module, 2018) extended the idea by computing channel attention followed by spatial attention, a map over image positions. ECA-Net (2019) argued that the dimensionality reduction in SE's excitation step hurts, and replaced the two fully connected layers with a single 1D convolution across neighboring channels, adding only 80 parameters to ResNet-50 against SE's roughly 2.5 million, while reporting a 2.28-point top-1 gain over the plain ResNet-50.

Compared with alternatives, SE's appeal is its cost-to-benefit ratio. Making a network deeper, such as going from ResNet-50 to ResNet-101, roughly doubles compute; adding SE blocks to ResNet-50 adds 0.26% more FLOPs for a similar accuracy gain. Spatial attention methods such as CBAM's second stage, or the trunk-and-mask attention of earlier residual attention networks, can capture where to look but cost more. Self-attention as used in vision transformers models relationships between all positions, which SE does not attempt; SE only models relationships between channels, using one global summary per channel.

The limitations are worth knowing. The parameter cost is not negligible: SE adds about 2.5 million parameters to ResNet-50's 25 million, a 10% increase, most of it in the last stage, which the authors show can be removed with less than 0.1% top-5 error cost, cutting the increase to 4%. Global average pooling throws away all spatial information, so SE cannot tell where in the image a feature fired. The excitation design is sensitive: replacing the sigmoid with ReLU made SE-ResNet-50 worse than the baseline ResNet-50. FLOPs understate real cost on some hardware, because the pooling and channel-wise multiplication add memory traffic; the official repository notes that the authors re-implemented global average pooling on GPU because the stock version was inefficient, and MobileNetV3 replaced the sigmoid partly for efficiency on phones.

04Esempio lavorato

Computing an SE block by hand on a tiny feature map

  1. Take a feature map with 4 channels, each 2 by 2 pixels, where channel averages work out to z = [2.0, 0.5, 1.0, 0.0].

  2. Choose a reduction ratio r = 2, so the first fully connected layer maps 4 values to 2 and the second maps 2 back to 4.

  3. Apply the first layer and a ReLU; with example weights the hidden vector might be [1.5, 0.0], since any negative result is clipped to zero.

  4. Apply the second layer and a sigmoid; example pre-activations of [2.0, -1.0, 0.5, -2.0] become weights of about [0.88, 0.27, 0.62, 0.12].

  5. Multiply every pixel in channel 1 by 0.88, channel 2 by 0.27, channel 3 by 0.62 and channel 4 by 0.12.

  6. Count the added parameters: 4 times 2 plus 2 times 4 equals 16 weights, matching the formula 2C squared divided by r.

Cosa mostra

A recalibrated feature map in which channel 1 is nearly unchanged and channel 4 is strongly suppressed for this input. Check it by implementing the same block in PyTorch with nn.AdaptiveAvgPool2d, two nn.Linear layers, ReLU and sigmoid, loading the same weights, and confirming the outputs match.

Implementazione nel mondo reale

The SENet ensemble that won the ILSVRC 2017 classification task reached a 2.251% top-5 error on the ImageNet test set, about 25% lower than the 2016 winner's 2.991%.

Adding SE blocks to ResNet-50 cut ImageNet top-5 validation error from 7.48% to 6.62% while raising compute from 3.86 to 3.87 GFLOPs, close to the much deeper ResNet-101.

Google's EfficientNet family uses the mobile inverted bottleneck block with squeeze-and-excitation added as the main building block of its baseline EfficientNet-B0 network.

MobileNetV3 places SE modules after the depthwise convolution, fixes their bottleneck at one quarter of the expansion channels, and swaps the sigmoid for a cheaper hard sigmoid for phones.

Using SE-ResNet-50 instead of ResNet-50 as the backbone of a Faster R-CNN detector raised COCO average precision by 2.4 points, a 6.3% relative improvement, in the SE paper's tests.

Il futuro delle reti di compressione ed eccitazione

Channel attention has become a routine design option rather than an active frontier, and most research since 2018 has refined the SE recipe. CBAM (2018) added a spatial attention stage, ECA-Net (2019) removed the dimensionality reduction to cut parameters to a handful, and mobile architectures such as MobileNetV3 and EfficientNet (both 2019) tuned the block's placement, width and activation for real hardware latency. Open questions remain practical. The SE authors suggested that a single reduction ratio throughout a network may not be optimal, and that more sophisticated squeeze operators than average pooling could help. Measured speed on specific devices often differs from FLOP counts, so architecture searches increasingly optimize latency directly, as the MobileNetV3 paper did. As vision transformers and hybrid convolution-attention models spread, the role of a separate channel-attention block in those designs is still being evaluated, and published ablations on standard benchmarks such as ImageNet and COCO remain the reliable way to judge whether SE-style blocks still pay for their cost in a given architecture.

Metterlo in pratica

Perché è importante

  • Costo e budget. Le decisioni relative all'architettura determinano prestazioni e costi operativi per anni.
  • Decisioni più chiare. La formazione tecnica aiuta i team a scegliere lo stack giusto, non solo quello più nuovo.
  • Controllo di qualità. Migliori scelte ingegneristiche riducono gli incidenti legati all’affidabilità nella produzione.

Attenzione

  • L'ottimizzazione di un benchmark può nascondere debolezze di sistema più ampie.
  • I costi delle infrastrutture e della manutenzione sono spesso sottostimati.
  • Le lacune in termini di sicurezza e osservabilità possono aumentare man mano che i sistemi diventano più complessi.

Prossimi passi

  1. Definire obiettivi di latenza, qualità e costi prima dell'implementazione.
  2. Benchmark in condizioni di carico e dati realistiche.
  3. Monitoraggio dello strumento per errori, deriva e impatto sull'utente.
  4. Preparare percorsi di rollback e risposta agli incidenti prima della scalabilità.

Fonti e approfondimenti

  1. arXiv (CVPR 2018)Reti di compressione ed eccitazione
  2. GitHub (hujie-frank)SENet: Squeeze-and-Excitation Networks (official Caffe implementation)
  3. arXiv (Google)Searching for MobileNetV3
  4. arXiv (Google)EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
  5. arXiv (KAIST, Lunit, Adobe Research)CBAM: Convolutional Block Attention Module
  6. arXivECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks

Continua a esplorare

Newsletter gratuita

Ricevi il briefing quotidiano sull'IA

Tre storie verificate sull'IA ogni mattina nei giorni feriali, scritte in un inglese semplice. Gratuito per sempre, senza pubblicità.

Un'email ogni giorno feriale. Disiscriviti con un clic. Non vendiamo né condividiamo mai il tuo indirizzo.

Mettiti alla prova

Take the Squeeze-and-Excitation Networks quiz

Feedback immediato su ogni risposta e un certificato condivisibile con un ID verificabile una volta superato un corso.

Inizia il quiz

Sostieni l'istruzione gratuita sull'IA. AI Understanding è un'organizzazione non profit 501(c)(3) — nessun annuncio, nessun paywall, mai. Fai una donazione

Domande frequenti

Che cos'è le Reti di Squeeze-and-Excitation?

A squeeze-and-excitation (SE) block is a small add-on for convolutional neural networks that summarizes each feature channel across the whole image, then learns a weight between 0 and 1 for every channel and rescales the channels accordingly. Introduced by Jie Hu and colleagues in 2017, SE networks won the ILSVRC 2017 image classification challenge and became a standard component of efficient vision models.

Cosa fa il passaggio 'squeeze' in un blocco SE?

La compressione applica il pooling medio globale, comprimendo ciascuna mappa di funzionalità in un singolo descrittore per canale.

Cosa produce il passo di 'eccitazione'?

L'eccitazione fa passare il vettore compresso attraverso due strati FC che terminano con un sigmoide, producendo pesi di ridimensionamento per canale.

Come vengono applicati i pesi di eccitazione alle mappe delle caratteristiche originali?

La mappa delle caratteristiche di ciascun canale viene moltiplicata per il suo peso appreso, amplificando o sopprimendo quel canale.

Perché l'eccitazione utilizza un collo di bottiglia con un rapporto di riduzione (spesso 16)?

Il collo di bottiglia si restringe quindi ripristina la dimensione del canale, quindi il blocco SE costa solo una piccola percentuale in più.

Quale competizione ha vinto SENet nel 2017?

SENet ha vinto la sfida di classificazione delle immagini ILSVRC 2017, portando l'errore dei primi 5 a circa il 2,25%.