Vision Transformers (ViTs) have achieved state-of-the-art performance on various computer vision applications. However, these models have considerable storage and computational overheads, making their deployment and efficient inference on edge devices challenging. Quantization is a promising approach to reducing model complexity, and the dyadic arithmetic pipeline can allow the quantized models to perform efficient integer-only inference. Unfortunately, dyadic arithmetic is based on the homogeneity condition in convolutional neural networks, which is not applicable to the non-linear components in ViTs, making integer-only inference of ViTs an open issue. In this paper, we propose I-ViT, an integer-only quantization scheme for ViTs, to enable ViTs to perform the entire computational graph of inference with integer arithmetic and bit-shifting, and without any floating-point arithmetic. In I-ViT, linear operations (e.g., MatMul and Dense) follow the integer-only pipeline with dyadic arithmetic, and non-linear operations (e.g., Softmax, GELU, and LayerNorm) are approximated by the proposed light-weight integer-only arithmetic methods. More specifically, I-ViT applies the proposed Shiftmax and ShiftGELU, which are designed to use integer bit-shifting to approximate the corresponding floating-point operations. We evaluate I-ViT on various benchmark models and the results show that integer-only INT8 quantization achieves comparable (or even slightly higher) accuracy to the full-precision (FP) baseline. Furthermore, we utilize TVM for practical hardware deployment on the GPU's integer arithmetic units, achieving 3.72$\sim$4.11$\times$ inference speedup compared to the FP model.

翻译：视觉Transformer（ViTs）已在多种计算机视觉应用中取得最先进性能。然而，这些模型具有可观的存储与计算开销，使其在边缘设备上的部署和高效推理面临挑战。量化是降低模型复杂度的有效途径，而二元算术流水线可使量化模型执行纯整数推理。遗憾的是，二元算术基于卷积神经网络中的同质性条件，并不适用于ViTs中的非线性组件，使得ViTs的纯整数推理成为开放性问题。本文提出I-ViT——一种面向ViTs的纯整数量化方案，使ViTs能够通过整数算术与位移操作完成整个推理计算图，完全无需浮点算术。在I-ViT中，线性操作（如MatMul和Dense）遵循二元算术的纯整数流水线，而非线性操作（如Softmax、GELU和LayerNorm）则通过所提出的轻量级纯整数算术方法进行近似。具体而言，I-ViT采用所提出的Shiftmax和ShiftGELU，它们通过整数位移操作来近似对应的浮点运算。我们在多种基准模型上评估I-ViT，结果表明纯整数INT8量化可达到与全精度（FP）基线相当（甚至略高）的准确率。此外，我们利用TVM在GPU整数算术单元上进行实际硬件部署，相比FP模型实现了3.72~4.11倍的推理加速。