Pretrained large language models (LLMs) exhibit exceptional general language processing capabilities but come with significant demands on memory and computational resources. As a powerful compression technology, binarization can extremely reduce model weights to a mere 1 bit, lowering the expensive computation and memory requirements. However, existing quantization techniques fall short of maintaining LLM performance under ultra-low bit-widths. In response to this challenge, we present BiLLM, a groundbreaking 1-bit post-training quantization scheme tailored for pretrained LLMs. Based on the weight distribution of LLMs, BiLLM first identifies and structurally selects salient weights, and minimizes the compression loss through an effective binary residual approximation strategy. Moreover, considering the bell-shaped distribution of the non-salient weights, we propose an optimal splitting search to group and binarize them accurately. BiLLM achieving for the first time high-accuracy inference (e.g. 8.41 perplexity on LLaMA2-70B) with only 1.08-bit weights across various LLMs families and evaluation metrics, outperforms SOTA quantization methods of LLM by significant margins. Moreover, BiLLM enables the binarization process of the LLM with 7 billion weights within 0.5 hours on a single GPU, demonstrating satisfactory time efficiency. Our code is available at https://github.com/Aaronhuang-778/BiLLM.

翻译：预训练大语言模型（LLMs）展现出卓越的通用语言处理能力，但伴随巨大的内存和计算资源需求。作为一种强大的压缩技术，二值化可将模型权重极端压缩至仅1比特，从而降低昂贵的计算与内存开销。然而，现有量化技术难以在超低比特宽度下维持LLM性能。针对这一挑战，我们提出BiLLM，一种为预训练LLM量身定制的开创性1比特训练后量化方案。基于LLM的权重分布特性，BiLLM首先识别并结构化选择显著权重，通过有效的二值残差逼近策略最小化压缩损失。此外，针对非显著权重的钟形分布，我们提出最优分裂搜索方法以实现精确分组二值化。BiLLM首次仅用1.08比特权重即可实现高精度推理（例如在LLaMA2-70B上困惑度达8.41），在各种LLM系列与评估指标上均显著超越当前最先进的LLM量化方法。同时，BiLLM可在单GPU上于0.5小时内完成70亿权重LLM的二值化过程，展现出令人满意的时间效率。我们的代码开源于https://github.com/Aaronhuang-778/BiLLM。