Mixup ak CutMix

Data augmentation creates new training examples from existing ones so a model sees more variety and memorizes less. Classic image augmentations transform one picture at a time: flip it, crop it, shift its colors. Mixup and CutMix take a different approach and combine two pictures, along with their labels, into one training example. The label is no longer a single class but a weighted mix, such as 60% dog and 40% cat. This encourages the network to make smooth, proportionate predictions rather than overconfident ones.

Mixup was introduced by Hongyi Zhang of MIT with Moustapha Cisse, Yann Dauphin and David Lopez-Paz of Facebook AI Research in "mixup: Beyond Empirical Risk Minimization" (arXiv October 2017, ICLR 2018). Empirical risk minimization (ERM) is the standard practice of minimizing average loss on the training examples themselves. Mixup instead trains on points in between examples. The procedure has four steps. First, draw a random mixing weight lambda from a Beta distribution with parameter alpha. Second, take two random training examples, often a batch and a shuffled copy of the same batch. Third, form the new input as lambda times the first input plus one minus lambda times the second. Fourth, form the new label the same way from the two one-hot label vectors, and train on that pair with ordinary cross-entropy loss. The authors note it takes a few lines of code and adds minimal computation.

The mixup paper reported gains across several domains. On ImageNet-2012, with 1.3 million training images in 1,000 classes, top-1 error for ResNet-50 trained for 200 epochs fell from 23.6% to 22.1%, and for ResNet-101 from 22.0% to 20.8%. The authors found that alpha between 0.1 and 0.4 helped on ImageNet while larger alpha caused underfitting, and that bigger models and longer training benefited most. They also showed mixup reduced memorization of deliberately corrupted labels, improved robustness to adversarial examples, and stabilized the training of generative adversarial networks.

CutMix came from Sangdoo Yun and colleagues at Naver's Clova AI Research and was published at ICCV 2019. It addressed a weakness the authors saw in two earlier methods. Cutout, from 2017, erases a random square of an image, forcing the model to look beyond the most obvious feature, but wastes those pixels. Mixup uses every pixel, but blended images look unnatural and locally ambiguous, and the CutMix paper reports that mixup lowered weakly supervised localization accuracy on ImageNet and object detection performance after transfer. CutMix replaces the erased rectangle with the matching patch from a second image and sets the label weights in proportion to the area each image occupies. The rectangle's size is drawn from the same Beta distribution, with alpha set to 1 so the area ratio is uniform between 0 and 1, and its position is drawn uniformly.

CutMix's results were strong across tasks. On ImageNet with ResNet-50, top-1 error fell from 23.68% to 21.40%, against 22.93% for Cutout, 22.58% for mixup and 22.50% for Manifold Mixup, a variant that mixes hidden-layer features. ResNet-101 improved by 1.70 percentage points. On CIFAR-100 it reached 14.47% top-1 error, a state-of-the-art result at the time. The paper also reported better weakly supervised object localization, gains when the pretrained network was fine-tuned for detection and captioning, improved robustness and reduced overconfidence.

Compared with alternatives, the two methods complement single-image augmentations rather than replacing them. Flips, crops and color jitter teach invariance to transformations of one image. Cutout and random erasing teach the model to use partial evidence. Mixup and CutMix additionally soften labels in a data-dependent way, which is related to label smoothing, a technique that replaces hard one-hot targets with slightly softened ones; the mixup paper's ablations found mixup beat combinations of input mixing and label smoothing. In practice, training recipes often apply both: libraries such as torchvision provide CutMix and MixUp transforms that can be randomly alternated per batch, and the DeiT recipe for vision transformers used both together with RandAugment and random erasing.

The methods have trade-offs. Because labels become fractional, training accuracy is no longer directly comparable to the usual metric, and the mixup authors found that longer training runs and larger models benefited most, so gains may be small on short schedules. Too strong a mix underfits, which is why alpha is tuned. Mixed images are not realistic, so mixup can hurt tasks that depend on precise spatial evidence, as the CutMix paper's localization and detection numbers show for mixup. CutMix can paste a patch that contains none of the second image's object, such as a patch of background, while the label still assigns that class weight in proportion to area. Both assume label mixing is meaningful, which is natural for classification but needs care for tasks such as segmentation or detection.

The cost is minimal. Both methods operate on a batch already in memory and add only a few tensor operations and a random draw per batch, so training time barely changes; the CutMix authors describe the overhead as negligible. The real cost is tuning: choosing alpha, the probability of applying each method, and the number of training epochs. Labels must be represented as probability vectors over classes rather than integer class indices, which torchvision's transforms handle by converting labels to a batch size by number of classes tensor.