With the popularity of the recent Transformer-based models represented by BERT, GPT-3 and ChatGPT, there has been state-of-the-art performance in a range of natural language processing tasks. However, the massive computations, huge memory footprint, and thus high latency of Transformer-based models is an inevitable challenge for the cloud with high real-time requirement. To tackle the issue, we propose BBCT, a method of block-wise bit-compression for transformer without retraining. Our method achieves more fine-grained compression of the whole transformer, including embedding, matrix multiplication, GELU, softmax, layer normalization, and all the intermediate results. As a case, we compress an efficient BERT with the method of BBCT. Our benchmark test results on General Language Understanding Evaluation (GLUE) show that BBCT can achieve less than 1% accuracy drop in most tasks.

翻译：随着以BERT、GPT-3和ChatGPT为代表的基于Transformer模型的普及，在自然语言处理的一系列任务中已取得了最先进的性能。然而，基于Transformer模型的大量计算、巨大内存占用以及由此带来的高延迟，对高实时要求的云端应用构成了不可避免的挑战。为解决这一问题，我们提出BBCT，一种无需重新训练的Transformer逐块比特压缩方法。该方法实现了对整个Transformer更细粒度的压缩，包括嵌入层、矩阵乘法、GELU、softmax、层归一化以及所有中间结果。作为实例，我们采用BBCT方法对高效的BERT模型进行压缩。在通用语言理解评估（GLUE）上的基准测试结果表明，BBCT在大多数任务中可实现低于1%的准确率下降。