Quantizing large language models has become a standard way to reduce their memory and computational costs. Typically, existing methods focus on breaking down the problem into individual layer-wise sub-problems, and minimizing per-layer error, measured via various metrics. Yet, this approach currently lacks theoretical justification and the metrics employed may be sub-optimal. In this paper, we present a "linearity theorem" establishing a direct relationship between the layer-wise $\ell_2$ reconstruction error and the model perplexity increase due to quantization. This insight enables two novel applications: (1) a simple data-free LLM quantization method using Hadamard rotations and MSE-optimal grids, dubbed HIGGS, which outperforms all prior data-free approaches such as the extremely popular NF4 quantized format, and (2) an optimal solution to the problem of finding non-uniform per-layer quantization levels which match a given compression constraint in the medium-bitwidth regime, obtained by reduction to dynamic programming. On the practical side, we demonstrate improved accuracy-compression trade-offs on Llama-3.1 and 3.2-family models, as well as on Qwen-family models. Further, we show that our method can be efficiently supported in terms of GPU kernels at various batch sizes, advancing both data-free and non-uniform quantization for LLMs.

翻译：大语言模型量化已成为降低其内存与计算成本的标准方法。现有方法通常将问题分解为逐层的子问题，并通过多种度量指标最小化每层误差。然而，该方法目前缺乏理论依据，且所采用的度量指标可能并非最优。本文提出一个"线性定理"，建立了逐层$\ell_2$重构误差与量化导致的模型困惑度上升之间的直接关系。这一洞见催生了两种新颖应用：(1) 一种利用哈达玛变换和均方误差最优网格的无数据LLM量化方法HIGGS，其性能超越所有先前的无数据方法（包括极受欢迎的NF4量化格式）；(2) 通过动态规划归约，在中位宽场景下实现了匹配给定压缩约束的非均匀逐层量化级别最优求解方案。在实践层面，我们在Llama-3.1/3.2系列模型及Qwen系列模型上展示了更优的精度-压缩权衡效果。此外，我们证明了该方法可在不同批量大小下通过GPU内核高效部署，从而推动了LLM无数据量化与非均匀量化的双重进展。