Modern pre-trained transformers have rapidly advanced the state-of-the-art in machine learning, but have also grown in parameters and computational complexity, making them increasingly difficult to deploy in resource-constrained environments. Binarization of the weights and activations of the network can significantly alleviate these issues, however is technically challenging from an optimization perspective. In this work, we identify a series of improvements which enables binary transformers at a much higher accuracy than what was possible previously. These include a two-set binarization scheme, a novel elastic binary activation function with learned parameters, and a method to quantize a network to its limit by successively distilling higher precision models into lower precision students. These approaches allow for the first time, fully binarized transformer models that are at a practical level of accuracy, approaching a full-precision BERT baseline on the GLUE language understanding benchmark within as little as 5.9%.

翻译：经过培训的现代变压器迅速提升了机器学习中的先进技术,但也提高了参数和计算复杂性,使其越来越难以在资源受限制的环境中部署。网络的重量和启动量的测算可以大大缓解这些问题,但从优化的角度看,在技术上具有挑战性。在这项工作中,我们确定了一系列改进措施,使二进制变压器的精度大大高于以前可能达到的精度。其中包括两套二进制计划、具有新颖的弹性二进制激活功能,以及将精度更高的模型连续推给低精度的学生,从而将一个网络量化到其极限的方法。这些方法首次允许将完全二进化的变压器模型在实际精确度上,接近GLUE语言理解基准的全精度BERT基线,在不到5.9%的范围内。