Deploying high-performance vision transformer (ViT) models on ubiquitous Internet of Things (IoT) devices to provide high-quality vision services will revolutionize the way we live, work, and interact with the world. Due to the contradiction between the limited resources of IoT devices and resource-intensive ViT models, the use of cloud servers to assist ViT model training has become mainstream. However, due to the larger number of parameters and floating-point operations (FLOPs) of the existing ViT models, the model parameters transmitted by cloud servers are large and difficult to run on resource-constrained IoT devices. To this end, this paper proposes a transmission-friendly ViT model, TFormer, for deployment on resource-constrained IoT devices with the assistance of a cloud server. The high performance and small number of model parameters and FLOPs of TFormer are attributed to the proposed hybrid layer and the proposed partially connected feed-forward network (PCS-FFN). The hybrid layer consists of nonlearnable modules and a pointwise convolution, which can obtain multitype and multiscale features with only a few parameters and FLOPs to improve the TFormer performance. The PCS-FFN adopts group convolution to reduce the number of parameters. The key idea of this paper is to propose TFormer with few model parameters and FLOPs to facilitate applications running on resource-constrained IoT devices to benefit from the high performance of the ViT models. Experimental results on the ImageNet-1K, MS COCO, and ADE20K datasets for image classification, object detection, and semantic segmentation tasks demonstrate that the proposed model outperforms other state-of-the-art models. Specifically, TFormer-S achieves 5% higher accuracy on ImageNet-1K than ResNet18 with 1.4$\times$ fewer parameters and FLOPs.

翻译：将高性能视觉Transformer（ViT）模型部署在无处不在的物联网（IoT）设备上，以提供高质量的视觉服务，将彻底改变我们的生活、工作以及与世界的交互方式。由于物联网设备资源有限与ViT模型资源密集之间的矛盾，利用云服务器辅助ViT模型训练已成为主流方法。然而，现有ViT模型参数数量与浮点运算次数（FLOPs）较大，导致云服务器传输的模型参数量庞大，难以在资源受限的物联网设备上运行。为此，本文提出一种传输友好的ViT模型TFormer，可在云服务器辅助下部署于资源受限的物联网设备。TFormer的高性能与较小的模型参数量及FLOPs，得益于所提出的混合层（hybrid layer）和部分连接前馈网络（PCS-FFN）。混合层由不可学习模块与逐点卷积组成，能够以极少的参数和FLOPs获取多类型、多尺度特征，从而提升TFormer性能。PCS-FFN采用分组卷积以减少参数量。本文的核心思想是提出参数量和FLOPs均较小的TFormer，以促进资源受限物联网设备上的应用从ViT模型的高性能中获益。在ImageNet-1K、MS COCO和ADE20K数据集上针对图像分类、目标检测和语义分割任务的实验结果表明，所提模型优于其他先进模型。具体而言，TFormer-S在ImageNet-1K上的准确率比ResNet18高出5%，且参数量和FLOPs减少1.4倍。