Vision Transformers (ViTs) have emerged as state-of-the-art models for various vision tasks recently. However, their heavy computation costs remain daunting for resource-limited devices. Consequently, researchers have dedicated themselves to compressing redundant information in ViTs for acceleration. However, they generally sparsely drop redundant image tokens by token pruning or brutally remove channels by channel pruning, leading to a sub-optimal balance between model performance and inference speed. They are also disadvantageous in transferring compressed models to downstream vision tasks that require the spatial structure of images, such as semantic segmentation. To tackle these issues, we propose a joint compression method for ViTs that offers both high accuracy and fast inference speed, while also maintaining favorable transferability to downstream tasks (CAIT). Specifically, we introduce an asymmetric token merging (ATME) strategy to effectively integrate neighboring tokens. It can successfully compress redundant token information while preserving the spatial structure of images. We further employ a consistent dynamic channel pruning (CDCP) strategy to dynamically prune unimportant channels in ViTs. Thanks to CDCP, insignificant channels in multi-head self-attention modules of ViTs can be pruned uniformly, greatly enhancing the model compression. Extensive experiments on benchmark datasets demonstrate that our proposed method can achieve state-of-the-art performance across various ViTs. For example, our pruned DeiT-Tiny and DeiT-Small achieve speedups of 1.7$\times$ and 1.9$\times$, respectively, without accuracy drops on ImageNet. On the ADE20k segmentation dataset, our method can enjoy up to 1.31$\times$ speedups with comparable mIoU. Our code will be publicly available.

翻译：视觉Transformer（ViTs）近年来已成为各类视觉任务的最先进模型。然而，其高昂的计算成本对资源受限设备而言仍构成严峻挑战。为此，研究人员致力于压缩ViT中的冗余信息以实现加速。但现有方法通常通过令牌剪枝稀疏丢弃冗余图像令牌，或通过通道剪枝粗暴移除通道，导致模型性能与推理速度之间难以达到最优平衡。此外，这些方法在将压缩模型迁移至需要图像空间结构的下游任务（如语义分割）时表现欠佳。针对上述问题，我们提出一种面向ViT的联合压缩方法，既能实现高精度与快速推理，又能保持对下游任务的良好可迁移性（CAIT）。具体而言，我们引入非对称令牌合并（ATME）策略，有效整合相邻令牌，在压缩冗余令牌信息的同时保留图像空间结构。同时采用一致动态通道剪枝（CDCP）策略，动态剪除ViT中不重要的通道。得益于CDCP，ViT多头自注意力模块中的次要通道可被统一剪枝，显著提升模型压缩效果。在基准数据集上的大量实验表明，本方法在多种ViT架构上均能取得最优性能。例如，经剪枝处理的DeiT-Tiny和DeiT-Small在ImageNet上分别实现1.7倍和1.9倍加速，且精度无损。在ADE20k分割数据集上，本方法可在保持可比mIoU的同时实现最高1.31倍加速。我们的代码将公开提供。