Network quantization significantly reduces model inference complexity and has been widely used in real-world deployments. However, most existing quantization methods have been developed mainly on Convolutional Neural Networks (CNNs), and suffer severe degradation when applied to fully quantized vision transformers. In this work, we demonstrate that many of these difficulties arise because of serious inter-channel variation in LayerNorm inputs, and present, Power-of-Two Factor (PTF), a systematic method to reduce the performance degradation and inference complexity of fully quantized vision transformers. In addition, observing an extreme non-uniform distribution in attention maps, we propose Log-Int-Softmax (LIS) to sustain that and simplify inference by using 4-bit quantization and the BitShift operator. Comprehensive experiments on various transformer-based architectures and benchmarks show that our Fully Quantized Vision Transformer (FQ-ViT) outperforms previous works while even using lower bit-width on attention maps. For instance, we reach 84.89% top-1 accuracy with ViT-L on ImageNet and 50.8 mAP with Cascade Mask R-CNN (Swin-S) on COCO. To our knowledge, we are the first to achieve lossless accuracy degradation (~1%) on fully quantized vision transformers. The code is available at https://github.com/megvii-research/FQ-ViT.

翻译：网络量化大幅降低了模型推理复杂度，并已在实际部署中得到广泛应用。然而，现有量化方法主要针对卷积神经网络（CNN）设计，当应用于全量化的视觉Transformer时会出现严重的性能退化。本工作证明，这些困难主要源于LayerNorm输入中显著的通道间差异，并提出了2的幂次因子（PTF）这一系统性方法，以减少全量化视觉Transformer的性能退化与推理复杂度。此外，针对注意力图中极端非均匀分布的现象，我们提出Log-Int-Softmax（LIS）以维持该分布特征，并通过使用4比特量化与BitShift算子简化推理流程。基于多种Transformer架构与基准的综合实验表明，我们的全量化视觉Transformer（FQ-ViT）在注意力图采用更低比特宽度的情况下仍优于此前工作。例如，在ImageNet上采用ViT-L达到84.89%的Top-1准确率，在COCO上采用Cascade Mask R-CNN（Swin-S）达到50.8 mAP。据我们所知，这是首个在全量化视觉Transformer上实现无损精度退化（约1%）的工作。代码开源于https://github.com/megvii-research/FQ-ViT。