Post-training quantization (PTQ) has driven attention to producing efficient large language models (LLMs) with ultra-low costs. Since hand-craft quantization parameters lead to low performance in low-bit quantization, recent methods optimize the quantization parameters through block-wise reconstruction between the floating-point and quantized models. However, these methods suffer from two challenges: accumulated errors from independent one-by-one block quantization and reconstruction difficulties from extreme weight and activation outliers. To address these two challenges, we propose CBQ, a cross-block reconstruction-based PTQ method for LLMs. To reduce error accumulation, we introduce a cross-block dependency with the aid of a homologous reconstruction scheme to build the long-range dependency between adjacent multi-blocks with overlapping. To reduce reconstruction difficulty, we design a coarse-to-fine pre-processing (CFP) to truncate weight outliers and dynamically scale activation outliers before optimization, and an adaptive rounding scheme, called LoRA-Rounding, with two low-rank learnable matrixes to further rectify weight quantization errors. Extensive experiments demonstrate that: (1) CBQ pushes both activation and weight quantization to low-bit settings W4A4, W4A8, and W2A16. (2) CBQ achieves better performance than the existing state-of-the-art methods on various LLMs and benchmark datasets.

翻译：后训练量化（PTQ）以超低成本实现高效大型语言模型（LLM）而备受关注。由于手工设计的量化参数在低位量化中性能较低，近期方法通过浮点模型与量化模型之间的逐块重构来优化量化参数。然而，这些方法面临两大挑战：独立逐块量化导致的累积误差，以及极端权重与激活异常值带来的重构困难。为解决上述问题，我们提出CBQ——一种基于跨块重构的LLM后训练量化方法。为减少误差累积，我们引入跨块依赖关系，结合同源重构方案构建具有重叠区域的相邻多块间长程依赖；为降低重构难度，我们设计粗到细预处理（CFP）策略，在优化前截断权重异常值并动态缩放激活异常值，同时提出名为LoRA-Rounding的自适应舍入方案，借助两个低秩可学习矩阵进一步修正权重量化误差。大量实验表明：（1）CBQ将激活与权重量化均推至W4A4、W4A8及W2A16等低位设置；（2）在多种大型语言模型与基准数据集上，CBQ性能均超越现有最优方法。