Vision transformers (ViTs) have significantly changed the computer vision landscape and have periodically exhibited superior performance in vision tasks compared to convolutional neural networks (CNNs). Although the jury is still out on which model type is superior, each has unique inductive biases that shape their learning and generalization performance. For example, ViTs have interesting properties with respect to early layer non-local feature dependence, as well as self-attention mechanisms which enhance learning flexibility, enabling them to ignore out-of-context image information more effectively. We hypothesize that this power to ignore out-of-context information (which we name $\textit{patch selectivity}$), while integrating in-context information in a non-local manner in early layers, allows ViTs to more easily handle occlusion. In this study, our aim is to see whether we can have CNNs $\textit{simulate}$ this ability of patch selectivity by effectively hardwiring this inductive bias using Patch Mixing data augmentation, which consists of inserting patches from another image onto a training image and interpolating labels between the two image classes. Specifically, we use Patch Mixing to train state-of-the-art ViTs and CNNs, assessing its impact on their ability to ignore out-of-context patches and handle natural occlusions. We find that ViTs do not improve nor degrade when trained using Patch Mixing, but CNNs acquire new capabilities to ignore out-of-context information and improve on occlusion benchmarks, leaving us to conclude that this training method is a way of simulating in CNNs the abilities that ViTs already possess. We will release our Patch Mixing implementation and proposed datasets for public use. Project page: https://arielnlee.github.io/PatchMixing/

翻译：视觉Transformer（ViT）已显著改变了计算机视觉领域格局，并在视觉任务中展现出相较于卷积神经网络（CNN）的周期性性能优势。尽管哪种模型类型更优尚无定论，但每种模型均具有独特的归纳偏置，这些偏置塑造了其学习与泛化性能。例如，ViT在早期层具有非局部特征依赖性的有趣特性，其自注意力机制增强了学习灵活性，使其能够更有效地忽略上下文无关的图像信息。我们假设，这种忽略上下文无关信息（我们称之为$\textit{块选择性}$）的能力，结合早期层以非局部方式整合上下文信息的能力，使得ViT能更轻松地处理遮挡问题。本研究旨在探究能否通过将这种归纳偏置有效硬编码到CNN中，使其$\textit{模拟}$ViT的块选择性能力。我们采用块混合数据增强方法——该方法将另一图像的块插入训练图像，并在两个图像类别间插值标签——来训练先进的ViT和CNN，评估其对模型忽略上下文无关块及处理自然遮挡能力的影响。研究发现，ViT在使用块混合训练时性能既未提升也未下降，但CNN获得了忽略上下文无关信息的新能力，并在遮挡基准测试中表现更优。因此我们得出结论，该训练方法是使CNN模拟ViT已有能力的有效途径。我们将公开块混合实现代码及所提出的数据集。项目页面：https://arielnlee.github.io/PatchMixing/