The rapid scaling of language models is motivating research using low-bitwidth quantization. In this work, we propose a novel binarization technique for Transformers applied to machine translation (BMT), the first of its kind. We identify and address the problem of inflated dot-product variance when using one-bit weights and activations. Specifically, BMT leverages additional LayerNorms and residual connections to improve binarization quality. Experiments on the WMT dataset show that a one-bit weight-only Transformer can achieve the same quality as a float one, while being 16x smaller in size. One-bit activations incur varying degrees of quality drop, but mitigated by the proposed architectural changes. We further conduct a scaling law study using production-scale translation datasets, which shows that one-bit weight Transformers scale and generalize well in both in-domain and out-of-domain settings. Implementation in JAX/Flax will be open sourced.

翻译：语言模型的快速扩展正推动着低比特量化研究。本文首次针对机器翻译任务提出一种新颖的Transformer二值化技术（BMT）。我们识别并解决了使用单比特权重和激活值时点积方差膨胀的问题。具体而言，BMT通过引入额外的层归一化和残差连接来提升二值化质量。在WMT数据集上的实验表明，单比特权重的Transformer可在体积缩小16倍的同时达到与浮点模型相同的质量。单比特激活值会导致不同程度的质量下降，但可通过所提出的架构改进加以缓解。我们进一步利用生产级翻译数据集进行了缩放定律研究，结果表明单比特权重Transformer在领域内和跨领域场景中均具有良好的缩放性与泛化能力。JAX/Flax实现代码将开源。