As the size of large language models (LLMs) continues to grow, model compression without sacrificing accuracy has become a crucial challenge for deployment. While some quantization methods, such as GPTQ, have made progress in achieving acceptable 4-bit weight-only quantization, attempts at lower bit quantization often result in severe performance degradation. In this paper, we introduce a technique called norm tweaking, which can be used as a plugin in current PTQ methods to achieve high precision while being cost-efficient. Our approach is inspired by the observation that rectifying the quantized activation distribution to match its float counterpart can readily restore accuracy for LLMs. To achieve this, we carefully design a tweaking strategy that includes calibration data generation and channel-wise distance constraint to update the weights of normalization layers for better generalization. We conduct extensive experiments on various datasets using several open-sourced LLMs. Our method demonstrates significant improvements in both weight-only quantization and joint quantization of weights and activations, surpassing existing PTQ methods. On GLM-130B and OPT-66B, our method even achieves the same level of accuracy at 2-bit quantization as their float ones. Our simple and effective approach makes it more practical for real-world applications.

翻译：随着大语言模型（LLMs）规模的持续增长，在不牺牲精度的前提下进行模型压缩已成为实际部署中的关键挑战。尽管GPTQ等量化方法在实现可接受的4位权重量化方面取得了进展，但尝试更低比特的量化往往会导致严重的性能退化。本文提出一种称为"norm tweaking"的技术，该技术可作为即插即用模块集成到现有后训练量化（PTQ）方法中，在保持低成本的同时实现高精度。我们的方法受以下观察启发：修正量化后的激活分布以匹配其浮点对应分布，即可轻松恢复LLMs的精度。为此，我们精心设计了包含校准数据生成和通道级距离约束的调整策略，用于更新归一化层的权重以提升泛化能力。我们使用多个开源LLMs在多种数据集上进行了广泛实验。在权重量化以及权重与激活联合量化任务中，我们的方法均展现出显著改进，超越了现有PTQ方法。针对GLM-130B和OPT-66B模型，本方法甚至在2位量化时达到了与浮点模型相同的精度水平。这种简单而有效的方法使其在实际应用中更具实用性。