The recently proposed Vision transformers (ViTs) have shown very impressive empirical performance in various computer vision tasks, and they are viewed as an important type of foundation model. However, ViTs are typically constructed with large-scale sizes, which then severely hinder their potential deployment in many practical resources-constrained applications. To mitigate this challenging problem, structured pruning is a promising solution to compress model size and enable practical efficiency. However, unlike its current popularity for CNNs and RNNs, structured pruning for ViT models is little explored. In this paper, we propose GOHSP, a unified framework of Graph and Optimization-based Structured Pruning for ViT models. We first develop a graph-based ranking for measuring the importance of attention heads, and the extracted importance information is further integrated to an optimization-based procedure to impose the heterogeneous structured sparsity patterns on the ViT models. Experimental results show that our proposed GOHSP demonstrates excellent compression performance. On CIFAR-10 dataset, our approach can bring 40% parameters reduction with no accuracy loss for ViT-Small model. On ImageNet dataset, with 30% and 35% sparsity ratio for DeiT-Tiny and DeiT-Small models, our approach achieves 1.65% and 0.76% accuracy increase over the existing structured pruning methods, respectively.

翻译：近期提出的视觉Transformer（Vision Transformers，ViTs）在多种计算机视觉任务中展现出极为优异的实证性能，被视为重要的一类基础模型。然而，ViT通常采用大规模参数构建，这严重制约了其在资源受限的实际应用场景中的部署潜力。为解决这一难题，结构化剪枝成为压缩模型尺寸并实现实际效率提升的有效途径。但不同于当前在CNN和RNN中的广泛适用性，针对ViT模型的结构化剪枝研究仍十分有限。本文提出GOHSP——一种面向ViT模型的图与优化融合结构化剪枝统一框架。我们首先开发基于图的重要性排序方法对注意力头进行重要性度量，随后将提取的重要性信息整合至基于优化的流程中，为ViT模型施加异构的结构化稀疏模式。实验结果表明，所提出的GOHSP展现出卓越的压缩性能。在CIFAR-10数据集上，该方法能在ViT-Small模型实现40%参数缩减且无精度损失；在ImageNet数据集上，针对DeiT-Tiny和DeiT-Small模型分别以30%和35%的稀疏比，相较于现有结构化剪枝方法分别实现1.65%和0.76%的精度提升。