With the advancement of diffusion models (DMs) and the substantially increased computational requirements, quantization emerges as a practical solution to obtain compact and efficient low-bit DMs. However, the highly discrete representation leads to severe accuracy degradation, hindering the quantization of diffusion models to ultra-low bit-widths. In this paper, we propose BinaryDM, a novel accurate quantization-aware training approach to push the weights of diffusion models towards the limit of 1-bit. Firstly, we present a Learnable Multi-basis Binarizer (LMB) to recover the representations generated by the binarized DM, which improves the information in details of representations crucial to the DM. Secondly, a Low-rank Representation Mimicking (LRM) is applied to enhance the binarization-aware optimization of the DM, alleviating the optimization direction ambiguity caused by fine-grained alignment. Moreover, a progressive initialization strategy is applied to training DMs to avoid convergence difficulties. Comprehensive experiments demonstrate that BinaryDM achieves significant accuracy and efficiency gains compared to SOTA quantization methods of DMs under ultra-low bit-widths. As the first binarization method for diffusion models, BinaryDM achieves impressive 16.0 times FLOPs and 27.1 times storage savings with 1-bit weight and 4-bit activation, showcasing its substantial advantages and potential for deploying DMs on resource-limited scenarios.

翻译：随着扩散模型（DMs）的进步以及计算需求的显著增加，量化作为一种获取紧凑高效的低比特扩散模型的实用方案应运而生。然而，高度离散化的表示导致了严重的精度退化，阻碍了扩散模型向超低比特宽度的量化。在本文中，我们提出BinaryDM，一种新颖的精确量化感知训练方法，旨在将扩散模型的权重推向1比特的极限。首先，我们提出了一种可学习多基二值化器（LMB），以恢复由二值化扩散模型生成的表示，从而改善对扩散模型至关重要的表示细节中的信息。其次，我们应用了低秩表示模仿（LRM）来增强扩散模型的二值化感知优化，缓解由细粒度对齐引起的优化方向模糊问题。此外，我们采用了一种渐进式初始化策略来训练扩散模型，以避免收敛困难。综合实验表明，与超低比特宽度下最先进的扩散模型量化方法相比，BinaryDM在精度和效率上均取得了显著提升。作为扩散模型的首个二值化方法，BinaryDM在1比特权重和4比特激活的情况下，实现了16.0倍的FLOPs和27.1倍的存储节省，展示了其在资源受限场景下部署扩散模型的显著优势和潜力。