The majority of quantization methods have been proposed to reduce the model size of Vision Transformers, yet most of them have overlooked the quantization of non-linear operations. Only a few works have addressed quantization for non-linear operations, but they applied a single quantization method across all non-linear operations. We believe that this can be further improved by employing a different quantization method for each non-linear operation. Therefore, to assign the most error-minimizing quantization method from the known methods to each non-linear layer, we propose a mixed non-linear quantization that considers layer-wise quantization sensitivity measured by SQNR difference metric. The results show that our method outperforms I-BERT, FQ-ViT, and I-ViT in both 8-bit and 6-bit settings for ViT, DeiT, and Swin models by an average of 0.6%p and 19.6%p, respectively. Our method outperforms I-BERT and I-ViT by 0.6%p and 20.8%p, respectively, when training time is limited. We plan to release our code at https://gitlab.com/ones-ai/mixed-non-linear-quantization.

翻译：大多数量化方法旨在减小视觉Transformer的模型尺寸，然而它们大多忽视了非线性操作的量化问题。仅有少数研究涉及非线性操作的量化，但这些工作对所有非线性操作统一应用了单一的量化方法。我们认为，通过为每种非线性操作采用不同的量化方法，可以进一步改进这一现状。因此，为了从已知方法中为每个非线性层分配误差最小化的量化方法，我们提出了一种混合非线性量化方法，该方法基于SQNR差异度量衡量的逐层量化敏感性进行选择。实验结果表明，在ViT、DeiT和Swin模型的8位和6位量化设置中，我们的方法平均分别优于I-BERT、FQ-ViT和I-ViT方法0.6%和19.6%。在训练时间受限的情况下，我们的方法分别优于I-BERT和I-ViT方法0.6%和20.8%。我们计划在https://gitlab.com/ones-ai/mixed-non-linear-quantization发布代码。