Quantization emerges as one of the most promising compression technologies for deploying efficient large models for various real time application in recent years. Considering that the storage and IO of weights take up the vast majority of the overhead inside a large model, weight only quantization can lead to large gains. However, existing quantization schemes suffer from significant accuracy degradation at very low bits, or require some additional computational overhead when deployed, making it difficult to be applied to large-scale applications in industry. In this paper, we propose decoupleQ, achieving a substantial increase in model accuracy, especially at very low bits. decoupleQ abandons the traditional heuristic quantization paradigm and decouples the model parameters into integer and floating-point parts, thus transforming the quantization problem into a traditional mathematical optimization problem with constraints, which is then solved alternatively by off-the-shelf optimization methods. Quantization via decoupleQ is linear and uniform, making it hardware-friendlier than non-uniform counterpart, and enabling the idea to be migrated to high-bit quantization to enhance its robustness. Our method has achieved well on-line accuracy near fp16/bf16 on the 2-bit quantization of large speech models in ByteDance. The code is available at https://github.com/bytedance/decoupleQ

翻译：量化是近年来为部署实时应用中的高效大模型而涌现出的最具前景的压缩技术之一。考虑到大模型中权重的存储与输入输出占据绝大部分开销，仅对权重进行量化可带来显著收益。然而，现有量化方案在极低比特下会遭遇严重精度损失，或是在部署时需要额外计算开销，因而难以应用于工业级大规模场景。本文提出decoupleQ方法，在极低比特下显著提升了模型精度。decoupleQ摒弃了传统启发式量化范式，将模型参数解耦为整数部分和浮点部分，从而将量化问题转化为含约束的传统数学优化问题，并利用现成的优化方法交替求解。采用decoupleQ的量化是线性且均匀的，相比非均匀量化更利于硬件实现，同时该思想可迁移至高比特量化以增强其鲁棒性。我们的方法在字节跳动大语音模型的2比特量化任务中取得了接近fp16/bf16水平的在线精度。代码已开源：https://github.com/bytedance/decoupleQ