The emergence of accurate open large language models (LLMs) has led to a race towards quantization techniques for such models enabling execution on end-user devices. In this paper, we revisit the problem of "extreme" LLM compression--defined as targeting extremely low bit counts, such as 2 to 3 bits per parameter, from the point of view of classic methods in Multi-Codebook Quantization (MCQ). Our work builds on top of Additive Quantization, a classic algorithm from the MCQ family, and adapts it to the quantization of language models. The resulting algorithm advances the state-of-the-art in LLM compression, outperforming all recently-proposed techniques in terms of accuracy at a given compression budget. For instance, when compressing Llama 2 models to 2 bits per parameter, our algorithm quantizes the 7B model to 6.93 perplexity (a 1.29 improvement relative to the best prior work, and 1.81 points from FP16), the 13B model to 5.70 perplexity (a .36 improvement) and the 70B model to 3.94 perplexity (a .22 improvement) on WikiText2. We release our implementation of Additive Quantization for Language Models AQLM as a baseline to facilitate future research in LLM quantization.

翻译：开源高精度大语言模型（LLMs）的涌现催生了面向此类模型的量化技术竞赛，旨在实现终端设备上的推理部署。本文从多码本量化（MCQ）经典方法的视角，重新审视了"极端"LLM压缩问题——即针对每个参数2至3比特的极低比特数目标。我们基于MCQ家族经典算法——加性量化（Additive Quantization）进行改进，并将其适配至语言模型的量化场景。该算法显著提升了LLM压缩的当前最优水平：在给定压缩预算下，其精度表现全面超越近期提出的各类技术方案。例如，将Llama 2模型压缩至每参数2比特时，本算法在WikiText2上的量化结果：7B模型达到6.93困惑度（相较此前最优方法提升1.29，与FP16差距缩小至1.81），13B模型为5.70困惑度（提升0.36），70B模型为3.94困惑度（提升0.22）。我们开源了面向语言模型的加性量化（AQLM）实现作为基准，以促进LLM量化领域的未来研究。