Large Language Models (LLMs) face significant deployment challenges due to their substantial memory requirements and the computational demands of auto-regressive text generation process. This paper addresses these challenges by focusing on the quantization of LLMs, a technique that reduces memory consumption by converting model parameters and activations into low-bit integers. We critically analyze the existing quantization approaches, identifying their limitations in balancing the accuracy and efficiency of the quantized LLMs. To advance beyond these limitations, we propose WKVQuant, a PTQ framework especially designed for quantizing weights and the key/value (KV) cache of LLMs. Specifically, we incorporates past-only quantization to improve the computation of attention. Additionally, we introduce two-dimensional quantization strategy to handle the distribution of KV cache, along with a cross-block reconstruction regularization for parameter optimization. Experiments show that WKVQuant achieves almost comparable memory savings to weight-activation quantization, while also approaching the performance of weight-only quantization.

翻译：大语言模型因其巨大的内存需求和自回归文本生成过程中的计算负荷而面临严峻的部署挑战。本文聚焦于大语言模型的量化技术——即将模型参数和激活值转换为低比特整数以降低内存消耗的方法，旨在攻克上述难题。我们深入剖析了现有量化方法，指出其在平衡量化后大语言模型的精度与效率方面存在的局限性。为突破这些局限，我们提出WKVQuant框架，这是一种专门用于量化大语言模型权重与键值缓存的训练后量化方法。具体而言，该框架引入仅对过去信息的量化策略以优化注意力计算；同时提出二维量化策略以适配键值缓存的分布特征，并结合跨块重建正则化方法优化参数更新。实验表明，WKVQuant在实现与权值-激活联合量化相近的内存节省效果的同时，其性能亦逼近纯权重量化方案。