Vision Transformers (ViTs) have demonstrated outstanding performance in computer vision tasks, yet their high computational complexity prevents their deployment in computing resource-constrained environments. Various token pruning techniques have been introduced to alleviate the high computational burden of ViTs by dynamically dropping image tokens. However, some undesirable pruning at early stages may result in permanent loss of image information in subsequent layers, consequently hindering model performance. To address this problem, we propose IdleViT, a dynamic token-idle-based method that achieves an excellent trade-off between performance and efficiency. Specifically, in each layer, IdleViT selects a subset of the image tokens to participate in computations while keeping the rest of the tokens idle and directly passing them to this layer's output. By allowing the idle tokens to be re-selected in the following layers, IdleViT mitigates the negative impact of improper pruning in the early stages. Furthermore, inspired by the normalized graph cut, we devise a token cut loss on the attention map as regularization to improve IdleViT's token selection ability. Our method is simple yet effective and can be extended to pyramid ViTs since no token is completely dropped. Extensive experimental results on various ViT architectures have shown that IdleViT can diminish the complexity of pretrained ViTs by up to 33\% with no more than 0.2\% accuracy decrease on ImageNet, after finetuning for only 30 epochs. Notably, when the keep ratio is 0.5, IdleViT outperforms the state-of-the-art EViT on DeiT-S by 0.5\% higher accuracy and even faster inference speed. The source code is available in the supplementary material.

翻译：视觉Transformer（Vision Transformers, ViTs）已在计算机视觉任务中展现出卓越性能，但其高计算复杂度阻碍了在计算资源受限环境中的部署。为缓解ViT的高计算负担，现有的各类令牌剪枝技术通过动态丢弃图像令牌来降低计算量。然而，早期阶段不当的剪枝可能导致后续层中图像信息的永久性丢失，从而阻碍模型性能。针对此问题，我们提出IdleViT——一种基于动态令牌闲置的方法，能在性能与效率之间实现出色平衡。具体而言，在每一层中，IdleViT选择部分图像令牌参与计算，同时将剩余令牌保持闲置状态并直接传递至该层输出。通过允许闲置令牌在后续层中被重新选择，IdleViT减轻了早期阶段不当剪枝的负面影响。此外，受归一化割（normalized graph cut）启发，我们在注意力图上设计了令牌割损失作为正则化项，以提升IdleViT的令牌选择能力。我们的方法简洁而有效，且可扩展至金字塔式ViT架构，因为没有任何令牌被完全丢弃。在多种ViT架构上的大量实验结果表明：在ImageNet数据集上，仅经30个微调周期后，IdleViT可将预训练ViT的计算复杂度降低多达33%，而准确率下降不超过0.2%。值得注意的是，当保留率为0.5时，IdleViT在DeiT-S上的准确率比当前最优的EViT方法高出0.5%，且推理速度更快。源代码见补充材料。