Recent years have witnessed the great success of vision transformer (ViT), which has achieved state-of-the-art performance on multiple computer vision benchmarks. However, ViT models suffer from vast amounts of parameters and high computation cost, leading to difficult deployment on resource-constrained edge devices. Existing solutions mostly compress ViT models to a compact model but still cannot achieve real-time inference. To tackle this issue, we propose to explore the divisibility of transformer structure, and decompose the large ViT into multiple small models for collaborative inference at edge devices. Our objective is to achieve fast and energy-efficient collaborative inference while maintaining comparable accuracy compared with large ViTs. To this end, we first propose a collaborative inference framework termed DeViT to facilitate edge deployment by decomposing large ViTs. Subsequently, we design a decomposition-and-ensemble algorithm based on knowledge distillation, termed DEKD, to fuse multiple small decomposed models while dramatically reducing communication overheads, and handle heterogeneous models by developing a feature matching module to promote the imitations of decomposed models from the large ViT. Extensive experiments for three representative ViT backbones on four widely-used datasets demonstrate our method achieves efficient collaborative inference for ViTs and outperforms existing lightweight ViTs, striking a good trade-off between efficiency and accuracy. For example, our DeViTs improves end-to-end latency by 2.89$\times$ with only 1.65% accuracy sacrifice using CIFAR-100 compared to the large ViT, ViT-L/16, on the GPU server. DeDeiTs surpasses the recent efficient ViT, MobileViT-S, by 3.54% in accuracy on ImageNet-1K, while running 1.72$\times$ faster and requiring 55.28% lower energy consumption on the edge device.

翻译：近年来，视觉Transformer（ViT）在多个计算机视觉基准测试中取得了最先进性能，展现了巨大成功。然而，ViT模型存在参数量庞大、计算成本高的问题，难以部署在资源受限的边缘设备上。现有解决方案大多将ViT模型压缩为紧凑模型，但仍无法实现实时推理。针对这一挑战，本文提出探索Transformer结构的可分性，将大型ViT分解为多个小模型以在边缘设备上进行协同推理。我们的目标是在保持与大型ViT相当精度的同时，实现快速且能效优化的协同推理。为此，我们首先提出名为DeViT的协同推理框架，通过分解大型ViT促进边缘部署。随后，设计基于知识蒸馏的分解-集成算法DEKD，融合多个小型分解模型并显著降低通信开销，同时开发特征匹配模块处理异构模型，促进分解模型对大型ViT的模仿。在四个广泛使用的数据集上，针对三种代表性ViT骨干网络的大量实验表明：本方法实现了ViT的高效协同推理，性能优于现有轻量级ViT，在效率与精度间取得良好平衡。例如，在GPU服务器上使用CIFAR-100数据集时，与大型ViT（ViT-L/16）相比，我们的DeViT仅牺牲1.65%精度便实现了2.89倍的端到端延迟提升；在ImageNet-1K数据集上，DeViT在边缘设备上的推理速度较最新高效ViT（MobileViT-S）提升1.72倍，能耗降低55.28%，同时精度提高3.54%。