Efficient deployment of large language models (LLMs) requires extreme quantization, forcing a critical trade-off between low-bit efficiency and performance. Residual binarization enables hardware-friendly, matmul-free inference by stacking binary ($\pm$1) layers, but is plagued by pathological feature co-adaptation. We identify a key failure mode, which we term inter-path adaptation: during quantization-aware training (QAT), parallel residual binary paths learn redundant features, degrading the error-compensation structure and limiting the expressive capacity of the model. While prior work relies on heuristic workarounds (e.g., path freezing) that constrain the solution space, we propose RaBiT, a novel quantization framework that resolves co-adaptation by algorithmically enforcing a residual hierarchy. Its core mechanism sequentially derives each binary path from a single shared full-precision weight, which ensures that every path corrects the error of the preceding one. This process is stabilized by a robust initialization that prioritizes functional preservation over mere weight approximation. RaBiT redefines the 2-bit accuracy-efficiency frontier: it achieves state-of-the-art performance, rivals even hardware-intensive Vector Quantization (VQ) methods, and delivers a $4.49\times$ inference speed-up over full-precision models on an RTX 4090. Code is available at https://github.com/SamsungLabs/RaBiT.

翻译：大语言模型的高效部署需要极端量化，这迫使我们在低位宽效率与性能之间进行关键权衡。残差二值化通过堆叠二值（±1）层实现了硬件友好、无需矩阵乘法的推理，但受到病态特征共适应的困扰。我们识别出一种关键失效模式，称为路径间适应：在量化感知训练过程中，并行残差二值路径学习了冗余特征，从而破坏了误差补偿结构并限制了模型的表达能力。虽然先前的工作依赖于启发式变通方法（例如路径冻结）来约束解空间，但我们提出了RaBiT，一种通过算法强制实施残差层次结构来解决共适应问题的新型量化框架。其核心机制从单一共享的全精度权重中顺序推导出每条二值路径，确保每条路径都能纠正前一条路径的误差。该过程通过优先考虑功能保留而非单纯权重近似的稳健初始化得到稳定。RaBiT重新定义了2比特精度-效率边界：它实现了最先进的性能，甚至可与硬件密集型的向量量化方法相媲美，并在RTX 4090上相较于全精度模型实现了4.49倍的推理加速。代码已开源：https://github.com/SamsungLabs/RaBiT。