Large Language Models (LLMs) have greatly pushed forward advancements in natural language processing, yet their high memory and computational demands hinder practical deployment. Binarization, as an effective compression technique, can shrink model weights to just 1 bit, significantly reducing the high demands on computation and memory. However, current binarization methods struggle to narrow the distribution gap between binarized and full-precision weights, while also overlooking the column deviation in LLM weight distribution. To tackle these issues, we propose ARB-LLM, a novel 1-bit post-training quantization (PTQ) technique tailored for LLMs. To narrow the distribution shift between binarized and full-precision weights, we first design an alternating refined binarization (ARB) algorithm to progressively update the binarization parameters, which significantly reduces the quantization error. Moreover, considering the pivot role of calibration data and the column deviation in LLM weights, we further extend ARB to ARB-X and ARB-RC. In addition, we refine the weight partition strategy with column-group bitmap (CGB), which further enhance performance. Equipping ARB-X and ARB-RC with CGB, we obtain ARB-LLM$_\text{X}$ and ARB-LLM$_\text{RC}$ respectively, which significantly outperform state-of-the-art (SOTA) binarization methods for LLMs. As a binary PTQ method, our ARB-LLM$_\text{RC}$ is the first to surpass FP16 models of the same size. The code and models will be available at https://github.com/ZHITENGLI/ARB-LLM.

翻译：大语言模型（LLMs）极大地推动了自然语言处理领域的进步，但其高昂的内存与计算需求阻碍了实际部署。二值化作为一种有效的模型压缩技术，可将模型权重压缩至仅1比特，显著降低对计算和内存的高要求。然而，现有二值化方法难以缩小二值化权重与全精度权重之间的分布差异，同时也忽视了大语言模型权重分布中存在的列偏差问题。为解决这些问题，我们提出了ARB-LLM，一种专为大语言模型设计的新型1比特训练后量化（PTQ）技术。为缩小二值化权重与全精度权重之间的分布偏移，我们首先设计了一种交替精炼二值化（ARB）算法，通过迭代更新二值化参数来显著降低量化误差。此外，考虑到校准数据的关键作用以及大语言模型权重中的列偏差，我们进一步将ARB扩展为ARB-X与ARB-RC。同时，我们采用列组位图（CGB）改进了权重划分策略，从而进一步提升性能。通过将ARB-X和ARB-RC与CGB相结合，我们分别得到了ARB-LLM$_\text{X}$和ARB-LLM$_\text{RC}$，其性能显著优于当前最先进（SOTA）的大语言模型二值化方法。作为一种二值化PTQ方法，我们的ARB-LLM$_\text{RC}$首次在同等规模下超越了FP16模型。代码与模型将在https://github.com/ZHITENGLI/ARB-LLM发布。