Vision transformers have demonstrated remarkable success in a wide range of computer vision tasks over the last years. However, their high computational costs remain a significant barrier to their practical deployment. In particular, the complexity of transformer models is quadratic with respect to the number of input tokens. Therefore techniques that reduce the number of input tokens that need to be processed have been proposed. This paper introduces Learned Thresholds token Merging and Pruning (LTMP), a novel approach that leverages the strengths of both token merging and token pruning. LTMP uses learned threshold masking modules that dynamically determine which tokens to merge and which to prune. We demonstrate our approach with extensive experiments on vision transformers on the ImageNet classification task. Our results demonstrate that LTMP achieves state-of-the-art accuracy across reduction rates while requiring only a single fine-tuning epoch, which is an order of magnitude faster than previous methods. Code is available at https://github.com/Mxbonn/ltmp .

翻译：近年来，视觉Transformer在各类计算机视觉任务中取得了显著成功。然而，其高昂的计算开销仍是实际部署的主要障碍，尤其是Transformer模型的计算复杂度与输入令牌数量呈二次方增长。为此，研究者提出了多种减少待处理输入令牌数量的技术。本文提出一种新颖的"学习阈值令牌合并与剪枝"（LTMP）方法，融合了令牌合并与令牌剪枝的双重优势。LTMP通过可学习的阈值掩码模块，动态决定哪些令牌应被合并、哪些应被剪枝。我们在ImageNet分类任务上对视觉Transformer进行了广泛实验，结果表明：LTMP在不同缩减率下均达到当前最优精度，且仅需一次微调迭代——其效率比现有方法快一个数量级。代码已开源至https://github.com/Mxbonn/ltmp。