技術指南

深度可分離卷積

A depthwise separable convolution splits an ordinary convolution layer into two cheaper steps: a depthwise step that filters each channel on its own, and a 1 by 1 pointwise step that mixes channels.

聆聽此指南AI 語音敘述 · ~8 min

本頁閱讀時間 9 分鐘
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. Counting the cost of one layer
  5. 現實世界的實施
  6. 深度可分離卷積的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

With 3 by 3 kernels it needs roughly 8 to 9 times less computation, which is why it underpins mobile vision networks such as MobileNet and Xception.

重點摘要

  1. A depthwise separable convolution replaces one standard convolution with a per-channel depthwise filter followed by a 1 by 1 pointwise convolution.
  2. Its cost relative to a standard convolution is 1/N plus 1/k squared, about one ninth for 3 by 3 kernels.
  3. MobileNet's depthwise separable version lost about 1 point of ImageNet accuracy while using roughly one eighth of the multiply-adds of a full-convolution version.
  4. Laurent Sifre developed the operation at Google Brain in 2013, and Xception and MobileNets popularized it in 2016 and 2017.
  5. Depthwise layers have high memory traffic per operation, so real speedups depend on the hardware and runtime, not just operation counts.

深入探討

A convolution layer in an image network slides small filters over a feature map, a grid of values with a height, a width and a number of channels. The input image has three channels, red, green and blue, and deeper layers commonly have hundreds of channels, each detecting some learned pattern. A standard convolution does two jobs at once. Each filter looks at a small spatial neighborhood, such as 3 by 3 pixels, and it looks at every input channel together, so a single filter both finds spatial patterns and combines channels. If there are M input channels and N output channels, the layer needs N separate filters, each of size 3 by 3 by M.

A depthwise separable convolution splits those two jobs. First, a depthwise convolution applies one small filter to each input channel separately, so channel 1 gets its own 3 by 3 filter, channel 2 gets its own, and so on, with no mixing between channels. Second, a pointwise convolution, which is a 1 by 1 convolution, combines the channels: at every pixel position it computes N weighted sums of the M depthwise outputs. The result has the same shape as a standard convolution's output, but the work is done in a cheaper, factorized way. The Xception paper notes that frameworks such as TensorFlow and Keras call this operation a separable convolution, which should not be confused with a spatially separable convolution that splits a 3 by 3 filter into 3 by 1 and 1 by 3 parts.

The savings follow from counting multiplications. For a feature map of size D_F by D_F, a standard convolution with a D_K by D_K kernel costs D_K x D_K x M x N x D_F x D_F multiply-adds. The depthwise step costs D_K x D_K x M x D_F x D_F, and the pointwise step costs M x N x D_F x D_F. Dividing the sum by the original gives 1/N + 1/D_K squared. With a 3 by 3 kernel and many output channels, that is just over one ninth, so the MobileNet paper describes the saving as 8 to 9 times less computation. Its worked example, an internal layer with 512 input and output channels on a 14 by 14 map, drops from 462 million multiply-adds and 2.36 million parameters to 52.3 million multiply-adds and 0.27 million parameters.

The idea has a short, well-documented history. According to the Xception paper, Laurent Sifre developed depthwise separable convolutions during a Google Brain internship in 2013, tested them in AlexNet, presented an overview at ICLR 2014 and reported detailed results in his thesis. Early Inception networks used one as their first layer. In October 2016 Francois Chollet posted Xception, which reinterpreted Inception modules as a step between ordinary and depthwise separable convolution and built an entire network from the latter. In April 2017 Andrew Howard and colleagues at Google posted MobileNets, which used depthwise separable layers everywhere except the first layer and added width and resolution multipliers to trade accuracy for speed. MobileNetV2, posted in January 2018, wrapped the depthwise step in an inverted residual block with a linear bottleneck. ConvNeXt, posted by Facebook AI Research in January 2022, showed that 7 by 7 depthwise convolutions help a modernized ResNet compete with vision transformers.

The accuracy cost is usually modest. MobileNet's own comparison found that a version built from full convolutions reached 71.7% ImageNet top-1 accuracy with 4,866 million multiply-adds and 29.3 million parameters, while the depthwise separable version reached 70.6% with 569 million multiply-adds and 4.2 million parameters: about 1 point of accuracy for roughly an eighth of the computation. Xception showed the reverse case: at a similar parameter count to Inception V3, building from depthwise separable layers improved ImageNet accuracy slightly and improved results more on Google's internal JFT dataset of 350 million images and 17,000 classes. Other alternatives to the standard convolution include grouped convolutions, which mix channels only within groups, and spatially separable kernels, but depthwise separable layers became the default building block of mobile vision models.

The main limitation is that fewer multiply-adds do not always mean faster code. The ShuffleNet V2 paper argued that speed also depends on memory access, and classified depthwise convolution with element-wise operations because it has a high ratio of memory access to computation. On hardware tuned for large matrix multiplications, such as data-center GPUs, a depthwise layer can therefore run well below the speed its operation count suggests. MobileNet's authors designed around this by putting 95% of computation time in 1 by 1 convolutions, which map directly to highly optimized matrix multiplication routines, with the depthwise layers holding about 3% of multiply-adds and 1% of parameters. A second limitation is capacity: depthwise filters have very few parameters, which is why MobileNet used little or no weight decay on them and MobileNetV2 expands channels before the depthwise step.

In practice, depthwise separable convolutions are free to use and built into every major framework. In PyTorch, a depthwise convolution is an ordinary Conv2d layer whose groups argument equals the number of input channels, followed by a Conv2d with a 1 by 1 kernel. TensorFlow and Keras offer it directly as a separable convolution layer. The main practical work is measuring real latency on the target device, because the theoretical saving only appears when the hardware and runtime handle depthwise layers efficiently.

04工作範例

Counting the cost of one layer

  1. Take a layer with a 14 by 14 feature map, 512 input channels, 512 output channels and a 3 by 3 kernel.

  2. Standard convolution: 3 x 3 x 512 x 512 x 14 x 14 is about 462 million multiply-adds, with 3 x 3 x 512 x 512, about 2.36 million weights.

  3. Depthwise step: 3 x 3 x 512 x 14 x 14 is about 0.9 million multiply-adds, with 4,608 weights.

  4. Pointwise step: 512 x 512 x 14 x 14 is about 51.4 million multiply-adds, with 262,144 weights.

  5. Add the two steps: about 52.3 million multiply-adds and about 0.27 million weights.

  6. Divide: 52.3 / 462 is about 0.113, matching 1/512 + 1/9.

它顯示了什麼

The learner reproduces the MobileNet paper's Table 3 figures, about 8.8 times fewer multiply-adds and parameters, and sees that the pointwise step now holds almost all of the cost, which is why fast 1 by 1 convolution matters for these networks.

現實世界的實施

On-phone image classification: MobileNetV2 reached 72.0% ImageNet top-1 accuracy with 3.4 million parameters and 300 million multiply-adds, running in about 75 milliseconds on one large core of a Google Pixel 1 phone with TensorFlow Lite.

Mobile object detection: the MobileNetV2 paper pairs the network with SSDLite, a detector that also uses depthwise separable layers, and reports it is 20 times more efficient and 10 times smaller than YOLOv2 while scoring higher on COCO.

Tiny models for constrained hardware: the original MobileNet paper's 0.25 width version has about 0.5 million parameters and 41 million multiply-adds, while still reaching 50.6% ImageNet accuracy.

Large-scale classification: Google's Xception replaced Inception modules with depthwise separable convolutions and, with slightly fewer parameters than Inception V3, scored 79.0% versus 78.2% top-1 on ImageNet.

Modern convolutional networks: ConvNeXt (2022), from Facebook AI Research, uses 7 by 7 depthwise convolutions to mix spatial information, separating spatial and channel mixing in a way its authors compare to vision transformers.

深度可分離卷積的未來

Depthwise separable convolutions remain a standard building block, but the questions around them have shifted from operation counts to measured speed. The ShuffleNet V2 work in 2018 showed that memory access and fragmentation can dominate runtime in lightweight networks, and the practical answer has been to design models with latency measured on the real target device rather than estimated from multiply-adds. The operation has also found a role beyond mobile models. The 2022 ConvNeXt study reported that moving to 7 by 7 depthwise kernels raised its model's ImageNet accuracy from 79.9% to 80.6% at roughly constant computation, with no further gain beyond 7 by 7, and its authors compared depthwise convolution to the per-channel weighted sum in self-attention. That framing, separate operations for spatial mixing and channel mixing, now appears in both convolutional and transformer-style designs. Open problems include making depthwise layers efficient on accelerators built for dense matrix multiplication, choosing kernel sizes and expansion ratios for each layer automatically, and deciding when large-kernel depthwise convolutions or attention give the better accuracy for a given hardware budget. Each new generation of mobile and edge hardware changes those trade-offs, so published results apply to the devices and runtimes they were measured on.

付諸實踐

為什麼這很重要

  • 成本與預算. 多年來,架構決策決定著效能和營運成本。
  • 更明確的決策. 技術教育幫助團隊選擇正確的堆疊,而不僅僅是最新的堆疊。
  • 品質管控. 更好的工程選擇可以減少生產中的可靠性事故。

留意

  • 優化一項基準測試可以隱藏更廣泛的系統弱點。
  • 基礎設施和維護成本常常被低估。
  • 隨著系統變得更加複雜,安全性和可觀察性差距可能會擴大。

後續步驟

  1. 在實施之前定義延遲、品質和成本目標。
  2. 在實際負載和資料條件下進行基準測試。
  3. 儀器監控錯誤、漂移和使用者影響。
  4. 在擴展之前準備回滾和事件回應路徑。

資料來源與延伸閱讀

  1. arXivMobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
  2. arXivXception: Deep Learning with Depthwise Separable Convolutions
  3. arXivMobileNetV2: Inverted Residuals and Linear Bottlenecks
  4. arXivShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
  5. arXivA ConvNet for the 2020s
  6. PyTorch documentationtorch.nn.Conv2d

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

Take the Depthwise Separable Convolutions quiz

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是深度可分離卷積?

A depthwise separable convolution splits an ordinary convolution layer into two cheaper steps: a depthwise step that filters each channel on its own, and a 1 by 1 pointwise step that mixes channels. With 3 by 3 kernels it needs roughly 8 to 9 times less computation, which is why it underpins mobile vision networks such as MobileNet and Xception.

深度可分離卷積將標準卷積分為哪兩個步驟?

深度步驟單獨對每個通道進行空間過濾,然後逐點 1x1 步驟組合跨通道的資訊。

在深度步驟中,如何處理輸入通道?

深度卷積將單獨的空間濾波器應用於每個輸入通道,而不混合通道,這使得它很便宜。

For many output channels, what operation-count reduction does the guide associate with a 3×3 depthwise separable convolution?

對於 3x3 內核,當輸出通道數量很大時,節省率接近 1/9,從而減少約 8 到 9 倍的操作。

在這個設計中,逐點 (1x1) 卷積的作用是什麼?

1x1 逐點卷積混合每個像素處的通道,而深度步驟則刻意避免這樣做。

哪個著名的架構在行動裝置上普及了深度可分離卷積?

MobileNet 是圍繞深度可分離卷積構建的,專門用於在手機上實現高效推理。