Weight quantization has become a standard tool for efficient LLM deployment, especially for local inference, where models are now routinely served at 2-3 bits per parameter. The state of the art is currently split into two sets of methods: simple scalar quantization techniques, such as GPTQ or AWQ, which are widely deployed but plateau in accuracy at 3-4 bits per parameter (bpp), and "second-generation" vector- or trellis-quantized methods, such as QTIP, GPTVQ and AQLM, which push the accuracy frontier at low bit-widths but are notoriously hard to implement and to scale, and have gained relatively less traction. In this paper, we ask whether this gap is fundamental, or whether a carefully optimized scalar quantizer can recover most of it. We answer in the affirmative, by introducing GSQ (Gumbel-Softmax Quantization), a post-training scalar quantization method which jointly learns the per-coordinate grid assignments and the per-group scales using a Gumbel-Softmax relaxation of the discrete grid. GSQ matches the cardinality of the relaxation to the small number of levels available in the target bit-width regime (e.g., 3-8 levels for ternary and 3 bpp, respectively), making the relaxation tight and the optimization tractable. Practically, on the standard Llama-3.1-8B/70B-Instruct models, GSQ closes most of the gap between scalar quantization and the QTIP frontier at 2 and 3 bits, while using a symmetric scalar grid with group-wise quantization, and thus fully compatible with existing scalar inference kernels. We further show that GSQ scales to trillion-scale Mixture-of-Experts models such as Kimi-K2.5, where vector-quantized methods are difficult to apply.

翻译：权重量化已成为高效部署大型语言模型（LLM）的标准工具，特别是在本地推理场景中，模型现在通常以每参数2-3比特的精度运行。当前最先进的方法分为两类：简单标量量化技术（如GPTQ或AWQ）虽被广泛部署，但在每参数3-4比特时精度停滞不前；而“第二代”向量量化或网格量化方法（如QTIP、GPTVQ和AQLM）虽在低位宽下推动了精度前沿，但因其实现和扩展难度极高而应用相对较少。本文旨在探究这一差距是根本性的，还是通过精心优化的标量量化器能够弥补大部分差距。我们给出的答案是肯定的，并引入GSQ（Gumbel-Softmax量化）——一种训练后标量量化方法，它通过Gumbel-Softmax对离散网格进行松弛，联合学习每坐标的网格分配与每组的缩放因子。GSQ将松弛的基数匹配为目标位宽区间内可用的少量级数（例如三值量化时为3个级，3比特时为8个级），从而使松弛更紧密且优化更易处理。在实际应用中，GSQ在标准Llama-3.1-8B/70B-Instruct模型上的2比特和3比特条件下，显著缩小了标量量化与QTIP前沿之间的差距，同时采用对称标量网格与分组量化，从而完全兼容现有标量推理内核。我们进一步证明，GSQ可扩展至万亿参数级别的混合专家模型（如Kimi-K2.5），而向量量化方法在此类模型上难以应用。