Affected by the massive amount of parameters, ViT usually suffers from serious overfitting problems with a relatively limited number of training samples. In addition, ViT generally demands heavy computing resources, which limit its deployment on resource-constrained devices. As a type of model-compression method,model binarization is potentially a good choice to solve the above problems. Compared with the full-precision one, the model with the binarization method replaces complex tensor multiplication with simple bit-wise binary operations and represents full-precision model parameters and activations with only 1-bit ones, which potentially solves the problem of model size and computational complexity, respectively. In this paper, we find that the decline of the accuracy of the binary ViT model is mainly due to the information loss of the Attention module and the Value vector. Therefore, we propose a novel model binarization technique, called Group Superposition Binarization (GSB), to deal with these issues. Furthermore, in order to further improve the performance of the binarization model, we have investigated the gradient calculation procedure in the binarization process and derived more proper gradient calculation equations for GSB to reduce the influence of gradient mismatch. Then, the knowledge distillation technique is introduced to alleviate the performance degradation caused by model binarization. Experiments on three datasets with limited numbers of training samples demonstrate that the proposed GSB model achieves state-of-the-art performance among the binary quantization schemes and exceeds its full-precision counterpart on some indicators.

翻译：受大量参数影响，ViT在训练样本相对有限时通常面临严重的过拟合问题。此外，ViT普遍需要大量计算资源，限制了其在资源受限设备上的部署。作为一种模型压缩方法，模型二值化是解决上述问题的潜在有效方案。与全精度模型相比，采用二值化方法的模型将复杂的张量乘法替换为简单的逐位二进制运算，并以1位数据表示全精度模型参数和激活值，分别解决了模型规模和计算复杂度问题。本文发现，二值化ViT模型准确率下降主要由注意力模块与值向量的信息损失导致。为此，我们提出一种新型模型二值化技术——组叠加二值化（GSB）以应对这些问题。此外，为进一步提升二值化模型性能，我们研究了二值化过程中的梯度计算流程，为GSB推导出更合理的梯度计算公式以减小梯度失配的影响。随后引入知识蒸馏技术缓解模型二值化导致的性能退化。在三个有限训练样本数据集上的实验表明，所提出的GSB模型在二值量化方案中取得最优性能，部分指标甚至超越其全精度对应模型。