Vision Transformer (ViT) has performed remarkably in various computer vision tasks. Nonetheless, affected by the massive amount of parameters, ViT usually suffers from serious overfitting problems with a relatively limited number of training samples. In addition, ViT generally demands heavy computing resources, which limit its deployment on resource-constrained devices. As a type of model-compression method, model binarization is potentially a good choice to solve the above problems. Compared with the full-precision one, the model with the binarization method replaces complex tensor multiplication with simple bit-wise binary operations and represents full-precision model parameters and activations with only 1-bit ones, which potentially solves the problem of model size and computational complexity, respectively. In this paper, we investigate a binarized ViT model. Empirically, we observe that the existing binarization technology designed for Convolutional Neural Networks (CNN) cannot migrate well to a ViT's binarization task. We also find that the decline of the accuracy of the binary ViT model is mainly due to the information loss of the Attention module and the Value vector. Therefore, we propose a novel model binarization technique, called Group Superposition Binarization (GSB), to deal with these issues. Furthermore, in order to further improve the performance of the binarization model, we have investigated the gradient calculation procedure in the binarization process and derived more proper gradient calculation equations for GSB to reduce the influence of gradient mismatch. Then, the knowledge distillation technique is introduced to alleviate the performance degradation caused by model binarization. Analytically, model binarization can limit the parameters search space during parameter updates while training a model....

翻译：视觉Transformer（ViT）在各类计算机视觉任务中表现卓越。然而，受海量参数影响，ViT在训练样本相对有限时通常面临严重的过拟合问题。此外，ViT普遍需要消耗大量计算资源，这限制了其在资源受限设备上的部署。作为一种模型压缩方法，模型二值化是解决上述问题的潜在良策。相较于全精度模型，采用二值化方法的模型通过简单的按位二进制运算替代复杂的张量乘法，并将全精度模型参数与激活值压缩为仅1比特表示，从而分别解决模型规模与计算复杂度问题。本文研究二值化ViT模型时发现：现有面向卷积神经网络（CNN）的二值化技术难以有效迁移至ViT的二值化任务；同时观察到二值化ViT模型精度下降主要源于注意力模块与Value向量的信息损失。为此，我们提出名为"群组叠加二值化"（GSB）的新型模型二值化技术应对上述挑战。为进一步提升二值化模型性能，我们深入探究二值化过程中的梯度计算机制，推导出更适配GSB的梯度方程以缓解梯度失配问题。此外，引入知识蒸馏技术以减轻模型二值化导致的性能退化。从理论分析角度，模型二值化可在训练过程中限制参数更新时的搜索空间...