Convolutions Kina Separable

A convolution layer in an image network slides small filters over a feature map, a grid of values with a height, a width and a number of channels. The input image has three channels, red, green and blue, and deeper layers commonly have hundreds of channels, each detecting some learned pattern. A standard convolution does two jobs at once. Each filter looks at a small spatial neighborhood, such as 3 by 3 pixels, and it looks at every input channel together, so a single filter both finds spatial patterns and combines channels. If there are M input channels and N output channels, the layer needs N separate filters, each of size 3 by 3 by M.

A depthwise separable convolution splits those two jobs. First, a depthwise convolution applies one small filter to each input channel separately, so channel 1 gets its own 3 by 3 filter, channel 2 gets its own, and so on, with no mixing between channels. Second, a pointwise convolution, which is a 1 by 1 convolution, combines the channels: at every pixel position it computes N weighted sums of the M depthwise outputs. The result has the same shape as a standard convolution's output, but the work is done in a cheaper, factorized way. The Xception paper notes that frameworks such as TensorFlow and Keras call this operation a separable convolution, which should not be confused with a spatially separable convolution that splits a 3 by 3 filter into 3 by 1 and 1 by 3 parts.

The savings follow from counting multiplications. For a feature map of size D_F by D_F, a standard convolution with a D_K by D_K kernel costs D_K x D_K x M x N x D_F x D_F multiply-adds. The depthwise step costs D_K x D_K x M x D_F x D_F, and the pointwise step costs M x N x D_F x D_F. Dividing the sum by the original gives 1/N + 1/D_K squared. With a 3 by 3 kernel and many output channels, that is just over one ninth, so the MobileNet paper describes the saving as 8 to 9 times less computation. Its worked example, an internal layer with 512 input and output channels on a 14 by 14 map, drops from 462 million multiply-adds and 2.36 million parameters to 52.3 million multiply-adds and 0.27 million parameters.

The idea has a short, well-documented history. According to the Xception paper, Laurent Sifre developed depthwise separable convolutions during a Google Brain internship in 2013, tested them in AlexNet, presented an overview at ICLR 2014 and reported detailed results in his thesis. Early Inception networks used one as their first layer. In October 2016 Francois Chollet posted Xception, which reinterpreted Inception modules as a step between ordinary and depthwise separable convolution and built an entire network from the latter. In April 2017 Andrew Howard and colleagues at Google posted MobileNets, which used depthwise separable layers everywhere except the first layer and added width and resolution multipliers to trade accuracy for speed. MobileNetV2, posted in January 2018, wrapped the depthwise step in an inverted residual block with a linear bottleneck. ConvNeXt, posted by Facebook AI Research in January 2022, showed that 7 by 7 depthwise convolutions help a modernized ResNet compete with vision transformers.

The accuracy cost is usually modest. MobileNet's own comparison found that a version built from full convolutions reached 71.7% ImageNet top-1 accuracy with 4,866 million multiply-adds and 29.3 million parameters, while the depthwise separable version reached 70.6% with 569 million multiply-adds and 4.2 million parameters: about 1 point of accuracy for roughly an eighth of the computation. Xception showed the reverse case: at a similar parameter count to Inception V3, building from depthwise separable layers improved ImageNet accuracy slightly and improved results more on Google's internal JFT dataset of 350 million images and 17,000 classes. Other alternatives to the standard convolution include grouped convolutions, which mix channels only within groups, and spatially separable kernels, but depthwise separable layers became the default building block of mobile vision models.

The main limitation is that fewer multiply-adds do not always mean faster code. The ShuffleNet V2 paper argued that speed also depends on memory access, and classified depthwise convolution with element-wise operations because it has a high ratio of memory access to computation. On hardware tuned for large matrix multiplications, such as data-center GPUs, a depthwise layer can therefore run well below the speed its operation count suggests. MobileNet's authors designed around this by putting 95% of computation time in 1 by 1 convolutions, which map directly to highly optimized matrix multiplication routines, with the depthwise layers holding about 3% of multiply-adds and 1% of parameters. A second limitation is capacity: depthwise filters have very few parameters, which is why MobileNet used little or no weight decay on them and MobileNetV2 expands channels before the depthwise step.

In practice, depthwise separable convolutions are free to use and built into every major framework. In PyTorch, a depthwise convolution is an ordinary Conv2d layer whose groups argument equals the number of input channels, followed by a Conv2d with a 1 by 1 kernel. TensorFlow and Keras offer it directly as a separable convolution layer. The main practical work is measuring real latency on the target device, because the theoretical saving only appears when the hardware and runtime handle depthwise layers efficiently.