CutMix is a popular augmentation technique commonly used for training modern convolutional and transformer vision networks. It was originally designed to encourage Convolution Neural Networks (CNNs) to focus more on an image's global context instead of local information, which greatly improves the performance of CNNs. However, we found it to have limited benefits for transformer-based architectures that naturally have a global receptive field. In this paper, we propose a novel data augmentation technique TokenMix to improve the performance of vision transformers. TokenMix mixes two images at token level via partitioning the mixing region into multiple separated parts. Besides, we show that the mixed learning target in CutMix, a linear combination of a pair of the ground truth labels, might be inaccurate and sometimes counter-intuitive. To obtain a more suitable target, we propose to assign the target score according to the content-based neural activation maps of the two images from a pre-trained teacher model, which does not need to have high performance. With plenty of experiments on various vision transformer architectures, we show that our proposed TokenMix helps vision transformers focus on the foreground area to infer the classes and enhances their robustness to occlusion, with consistent performance gains. Notably, we improve DeiT-T/S/B with +1% ImageNet top-1 accuracy. Besides, TokenMix enjoys longer training, which achieves 81.2% top-1 accuracy on ImageNet with DeiT-S trained for 400 epochs. Code is available at https://github.com/Sense-X/TokenMix.

翻译：CutMix是一种常用于训练现代卷积与Transformer视觉网络的增强技术。其初衷是激励卷积神经网络（CNNs）更多关注图像的全局上下文而非局部信息，从而大幅提升CNNs性能。然而，我们发现该方法对天然具有全局感受野的Transformer架构增益有限。本文提出一种新型数据增强技术TokenMix，旨在提升视觉Transformer的性能。TokenMix通过将混合区域分割为多个独立子区域，在令牌（token）层面实现两幅图像的混合。此外，我们指出CutMix中混合学习目标——即一对真实标签的线性组合——可能不精确且有时有悖直觉。为获得更合适的目标，我们提出基于预训练教师模型对两幅图像的内容相关神经激活图分配目标分数，且该教师模型无需具备高性能。通过在多种视觉Transformer架构上的大量实验，我们证明TokenMix能帮助视觉Transformer聚焦前景区域以推断类别，并增强其对遮挡的鲁棒性，且性能提升稳定。值得注意的是，我们在DeiT-T/S/B上取得了ImageNet top-1准确率+1%的提升。此外，TokenMix支持更长时间训练，当DeiT-S训练400个epoch时，在ImageNet上实现了81.2%的top-1准确率。代码已开源：https://github.com/Sense-X/TokenMix。