Recently, Vision Transformers (ViTs) have attracted a lot of attention in the field of computer vision. Generally, the powerful representative capacity of ViTs mainly benefits from the self-attention mechanism, which has a high computation complexity. To accelerate ViTs, we propose an integrated compression pipeline based on observed heterogeneous attention patterns across layers. On one hand, different images share more similar attention patterns in early layers than later layers, indicating that the dynamic query-by-key self-attention matrix may be replaced with a static self-attention matrix in early layers. Then, we propose a dynamic-guided static self-attention (DGSSA) method where the matrix inherits self-attention information from the replaced dynamic self-attention to effectively improve the feature representation ability of ViTs. On the other hand, the attention maps have more low-rank patterns, which reflect token redundancy, in later layers than early layers. In a view of linear dimension reduction, we further propose a method of global aggregation pyramid (GLAD) to reduce the number of tokens in later layers of ViTs, such as Deit. Experimentally, the integrated compression pipeline of DGSSA and GLAD can accelerate up to 121% run-time throughput compared with DeiT, which surpasses all SOTA approaches.

翻译：近期，视觉Transformer（ViT）在计算机视觉领域引起了广泛关注。通常，ViT强大的表征能力主要得益于具有高计算复杂度的自注意力机制。为了加速ViT，我们基于跨层的异构注意力模式观察，提出了一种集成压缩流水线。一方面，与深层相比，不同图像在浅层共享更相似的注意力模式，这表明浅层中动态的逐查询（query-by-key）自注意力矩阵可被静态自注意力矩阵替代。据此，我们提出了一种动态引导的静态自注意力（DGSSA）方法，其中矩阵继承了被替代动态自注意力的信息，从而有效提升ViT的特征表示能力。另一方面，与浅层相比，深层注意力图具有更多反映令牌冗余的低秩模式。从线性降维角度出发，我们进一步提出全局聚合金字塔（GLAD）方法，以减少ViT（如DeiT）中深层的令牌数量。实验表明，与DeiT相比，DGSSA与GLAD的集成压缩流水线可实现最高121%的运行时吞吐量提升，超越了所有现有最优方法。