Large language models (LLMs) excel in natural language processing but demand intensive computation. To mitigate this, various quantization methods have been explored, yet they compromise LLM performance. This paper unveils a previously overlooked type of outlier in LLMs. Such outliers are found to allocate most of the attention scores on initial tokens of input, termed as pivot tokens, which is crucial to the performance of quantized LLMs. Given that, we propose IntactKV to generate the KV cache of pivot tokens losslessly from the full-precision model. The approach is simple and easy to combine with existing quantization solutions. Besides, IntactKV can be calibrated as additional LLM parameters to boost the quantized LLMs further. Mathematical analysis also proves that IntactKV effectively reduces the upper bound of quantization error. Empirical results show that IntactKV brings consistent improvement and achieves lossless weight-only INT4 quantization on various downstream tasks, leading to the new state-of-the-art for LLM quantization.

翻译：大语言模型（LLMs）在自然语言处理中表现出色，但计算量巨大。为缓解此问题，已有多种量化方法被探索，但常以牺牲模型性能为代价。本文揭示了LLMs中一种被先前研究忽视的异常值类型。此类异常值将大部分注意力分数分配给输入中的初始词元（称为中枢词元），这对量化后LLM的性能至关重要。基于此，我们提出IntactKV方法，从全精度模型中无损生成中枢词元的KV缓存。该方法简洁易用，可集成现有量化方案。此外，IntactKV可作为可校准的额外LLM参数，进一步提升量化模型性能。数学分析证明，IntactKV有效降低了量化误差上界。实验结果表明，IntactKV能带来持续改进，在多种下游任务上实现无损的仅权重量化（INT4），刷新了LLM量化的最新水平。