Attention-based vision models, such as Vision Transformer (ViT) and its variants, have shown promising performance in various computer vision tasks. However, these emerging architectures suffer from large model sizes and high computational costs, calling for efficient model compression solutions. To date, pruning ViTs has been well studied, while other compression strategies that have been widely applied in CNN compression, e.g., model factorization, is little explored in the context of ViT compression. This paper explores an efficient method for compressing vision transformers to enrich the toolset for obtaining compact attention-based vision models. Based on the new insight on the multi-head attention layer, we develop a highly efficient ViT compression solution, which outperforms the state-of-the-art pruning methods. For compressing DeiT-small and DeiT-base models on ImageNet, our proposed approach can achieve 0.45% and 0.76% higher top-1 accuracy even with fewer parameters. Our finding can also be applied to improve the customization efficiency of text-to-image diffusion models, with much faster training (up to $2.6\times$ speedup) and lower extra storage cost (up to $1927.5\times$ reduction) than the existing works.

翻译：基于注意力的视觉模型，如视觉Transformer（ViT）及其变体，已在各类计算机视觉任务中展现出优异性能。然而，这些新兴架构存在模型规模庞大和计算成本高昂的问题，亟需高效的模型压缩方案。目前，ViT剪枝技术已被广泛研究，但其他在 CNN 压缩中普遍应用的压缩策略（如模型分解）在 ViT 压缩领域仍鲜有探索。本文探索了一种高效的视觉Transformer压缩方法，以丰富紧凑型注意力视觉模型的构建工具集。基于对多头注意力层的新认知，我们开发了一种高性能 ViT 压缩方案，其性能优于当前最先进的剪枝方法。在 ImageNet 上压缩 DeiT-small 和 DeiT-base 模型时，本方法即便在参数更少的情况下，仍能分别实现 0.45% 和 0.76% 的 top-1 准确率提升。此外，该发现还可用于提升文生图扩散模型的定制效率：与现有工作相比，训练速度最高提升 2.6 倍，额外存储成本最高降低 1927.5 倍。