This work studies post-training parameter quantization in large language models (LLMs). We introduce quantization with incoherence processing (QuIP), a new method based on the insight that quantization benefits from $\textit{incoherent}$ weight and Hessian matrices, i.e., from the weights being even in magnitude and the directions in which it is important to round them accurately being unaligned with the coordinate axes. QuIP consists of two steps: (1) an adaptive rounding procedure minimizing a quadratic proxy objective; (2) efficient pre- and post-processing that ensures weight and Hessian incoherence via multiplication by random orthogonal matrices. We complement QuIP with the first theoretical analysis for an LLM-scale quantization algorithm, and show that our theory also applies to an existing method, OPTQ. Empirically, we find that our incoherence preprocessing improves several existing quantization algorithms and yields the first LLM quantization methods that produce viable results using only two bits per weight. Our code can be found at https://github.com/Cornell-RelaxML/QuIP.

翻译：本文研究了大语言模型（LLMs）中的训练后参数量化问题。我们提出了一种基于非相干处理（QuIP）的量化方法，该方法基于量化受益于$\textit{非相干}$的权重和Hessian矩阵这一洞见——即权重大小均匀分布，且需要精确舍入的方向与坐标轴非对齐。QuIP包含两个步骤：（1）一种最小化二次代理目标的自适应舍入过程；（2）通过乘以随机正交矩阵确保权重和Hessian矩阵非相干性的高效预处理与后处理。我们为QuIP补充了首个针对LLM规模量化算法的理论分析，并证明该理论同样适用于现有方法OPTQ。实验表明，我们的非相干预处理改进多种现有量化算法，并首次实现仅需每权重两比特即可产生可行结果的LLM量化方法。我们的代码发布在https://github.com/Cornell-RelaxML/QuIP。