As the size and context length of Large Language Models (LLMs) grow, weight-activation quantization has emerged as a crucial technique for efficient deployment of LLMs. Compared to weight-only quantization, weight-activation quantization presents greater challenges due to the presence of outliers in activations. Existing methods have made significant progress by exploring mixed-precision quantization and outlier suppression. However, these methods primarily focus on optimizing the results of single matrix multiplication, neglecting the bidirectional propagation of quantization errors in LLMs. Specifically, errors accumulate vertically within the same token through layers, and diffuse horizontally across different tokens due to self-attention mechanisms. To address this issue, we introduce BiSup, a Bidirectional quantization error Suppression method. By constructing appropriate optimizable parameter spaces, BiSup utilizes a small amount of data for quantization-aware parameter-efficient fine-tuning to suppress the error vertical accumulation. Besides, BiSup employs prompt mixed-precision quantization strategy, which preserves high precision for the key-value cache of system prompts, to mitigate the error horizontal diffusion. Extensive experiments on Llama and Qwen families demonstrate that BiSup can improve performance over two state-of-the-art methods (the average WikiText2 perplexity decreases from 13.26 to 9.41 for Atom and from 14.33 to 7.85 for QuaRot under the W3A3-g128 configuration), further facilitating the practical applications of low-bit weight-activation quantization.

翻译：随着大语言模型（LLM）的规模与上下文长度不断增长，权重-激活值量化已成为高效部署LLM的关键技术。与仅权重量化相比，由于激活值中存在离群值，权重-激活值量化面临更大挑战。现有方法通过探索混合精度量化与离群值抑制已取得显著进展。然而，这些方法主要聚焦于优化单次矩阵乘法的结果，忽视了量化误差在LLM中的双向传播。具体而言，误差在同一令牌内通过各网络层垂直累积，并因自注意力机制在不同令牌间水平扩散。为解决此问题，我们提出BiSup，一种双向量化误差抑制方法。通过构建合适的可优化参数空间，BiSup利用少量数据进行量化感知的参数高效微调，以抑制误差的垂直累积。此外，BiSup采用提示词混合精度量化策略，对系统提示词的键值缓存保持高精度，以减轻误差的水平扩散。在Llama和Qwen系列模型上的大量实验表明，BiSup的性能优于两种前沿方法（在W3A3-g128配置下，WikiText2困惑度平均值从Atom的13.26降至9.41，从QuaRot的14.33降至7.85），进一步推动了低位宽权重-激活值量化的实际应用。