With the popularity of the recent Transformer-based models represented by BERT, GPT-3 and ChatGPT, there has been state-of-the-art performance in a range of natural language processing tasks. However, the massive computations, huge memory footprint, and thus high latency of Transformer-based models is an inevitable challenge for the cloud with high real-time requirement. To tackle the issue, we propose BBCT, a method of block-wise bit-compression for transformer without retraining. Our method achieves more fine-grained compression of the whole transformer, including embedding, matrix multiplication, GELU, softmax, layer normalization, and all the intermediate results. As a case, we compress an efficient BERT with the method of BBCT. Our benchmark test results on General Language Understanding Evaluation (GLUE) show that BBCT can achieve less than 1% accuracy drop in most tasks.

翻译：随着以BERT、GPT-3和ChatGPT为代表的Transformer模型近年来的普及，它们在自然语言处理任务中取得了最先进的性能。然而，Transformer模型庞大的计算量、巨大的内存占用以及由此产生的高延迟，给具有高实时性需求的云端应用带来了不可避免的挑战。为解决这一问题，我们提出了BBCT，一种无需重新训练的Transformer逐块位压缩方法。我们的方法实现了对整个Transformer（包括嵌入层、矩阵乘法、GELU激活函数、Softmax、层归一化以及所有中间结果）的更细粒度压缩。以案例研究为例，我们采用BBCT方法对高效的BERT模型进行了压缩。在通用语言理解评估（GLUE）基准测试上的结果表明，BBCT在大多数任务上的准确率下降幅度小于1%。