Deploying large language models (LLMs) often faces challenges from substantial memory and computational costs. Quantization offers a solution, yet performance degradation in the sub-1-bit regime remains particularly difficult. This paper introduces LittleBit, a novel method for extreme LLM compression. It targets levels like 0.1 bits per weight (BPW), achieving nearly 31$\times$ memory reduction, e.g., Llama2-13B to under 0.9 GB. LittleBit represents weights in a low-rank form using latent matrix factorization, subsequently binarizing these factors. To counteract information loss from this extreme precision, it integrates a multi-scale compensation mechanism. This includes row, column, and an additional latent dimension that learns per-rank importance. Two key contributions enable effective training: Dual Sign-Value-Independent Decomposition (Dual-SVID) for quantization-aware training (QAT) initialization, and integrated Residual Compensation to mitigate errors. Extensive experiments confirm LittleBit's superiority in sub-1-bit quantization: e.g., its 0.1 BPW performance on Llama2-7B surpasses the leading method's 0.7 BPW. LittleBit establishes a new, viable size-performance trade-off--unlocking a potential 11.6$\times$ speedup over FP16 at the kernel level--and makes powerful LLMs practical for resource-constrained environments. Our code can be found at https://github.com/SamsungLabs/LittleBit.

翻译：部署大型语言模型（LLMs）常面临巨大的内存和计算成本挑战。量化提供了一种解决方案，但在低于1比特的量化区间内，性能下降问题尤为突出。本文提出LittleBit，一种用于极端LLM压缩的新方法。该方法以0.1比特每权重（BPW）等超低位宽为目标，实现了近31倍的内存压缩，例如将Llama2-13B模型压缩至0.9 GB以下。LittleBit利用隐矩阵分解将权重表示为低秩形式，随后对这些因子进行二值化。为了抵消这种极端精度带来的信息损失，该方法集成了一个多尺度补偿机制，包括行、列以及一个学习各秩重要性的额外隐维度。两项关键贡献确保了训练的有效性：用于量化感知训练（QAT）初始化的双符号-值独立分解（Dual-SVID），以及用于减少误差的集成残差补偿。大量实验证实了LittleBit在低于1比特量化中的优越性：例如，其在Llama2-7B模型上的0.1 BPW性能超越了领先方法的0.7 BPW性能。LittleBit确立了一种新颖且可行的尺寸-性能权衡方案——在核心层面实现了相对于FP16高达11.6倍的潜在加速比——使得强大的LLM能够在资源受限的环境中实际部署。我们的代码可在 https://github.com/SamsungLabs/LittleBit 获取。