Although vision transformers (ViTs) have shown promising results in various computer vision tasks recently, their high computational cost limits their practical applications. Previous approaches that prune redundant tokens have demonstrated a good trade-off between performance and computation costs. Nevertheless, errors caused by pruning strategies can lead to significant information loss. Our quantitative experiments reveal that the impact of pruned tokens on performance should be noticeable. To address this issue, we propose a novel joint Token Pruning & Squeezing module (TPS) for compressing vision transformers with higher efficiency. Firstly, TPS adopts pruning to get the reserved and pruned subsets. Secondly, TPS squeezes the information of pruned tokens into partial reserved tokens via the unidirectional nearest-neighbor matching and similarity-based fusing steps. Compared to state-of-the-art methods, our approach outperforms them under all token pruning intensities. Especially while shrinking DeiT-tiny&small computational budgets to 35%, it improves the accuracy by 1%-6% compared with baselines on ImageNet classification. The proposed method can accelerate the throughput of DeiT-small beyond DeiT-tiny, while its accuracy surpasses DeiT-tiny by 4.78%. Experiments on various transformers demonstrate the effectiveness of our method, while analysis experiments prove our higher robustness to the errors of the token pruning policy. Code is available at https://github.com/megvii-research/TPS-CVPR2023.

翻译：尽管视觉Transformer（ViTs）近年来在多种计算机视觉任务中展现出令人瞩目的成果，但其高计算成本限制了实际应用。先前通过剪枝冗余令牌的方法在性能与计算开销之间取得了良好平衡，然而，剪枝策略导致的误差可能引发显著的信息损失。我们的定量实验表明，被剪除令牌对性能的影响不容忽视。为解决该问题，我们提出一种新颖的联合令牌剪枝与挤压模块（TPS），以实现更高效率的视觉Transformer压缩。首先，TPS通过剪枝获得保留子集与剪除子集；其次，TPS通过单向最近邻匹配与基于相似性的融合步骤，将剪除令牌的信息压缩至部分保留令牌中。与现有最优方法相比，我们的方法在所有令牌剪枝强度下均表现更优。特别地，当DeiT-tiny与DeiT-small的计算预算缩减至35%时，在ImageNet分类任务上，我们的方法相比基线模型将准确率提升了1%-6%。该方法可加速DeiT-small的吞吐量超越DeiT-tiny，同时其准确率比DeiT-tiny高出4.78%。针对多种Transformer的实验验证了本方法的有效性，而分析实验进一步证明，本方法对令牌剪枝策略的误差具有更强的鲁棒性。代码已开源至https://github.com/megvii-research/TPS-CVPR2023。