The LoRA-finetuning quantization of LLMs has been extensively studied to obtain accurate yet compact LLMs for deployment on resource-constrained hardware. However, existing methods cause the quantized LLM to severely degrade and even fail to benefit from the finetuning of LoRA. This paper proposes a novel IR-QLoRA for pushing quantized LLMs with LoRA to be highly accurate through information retention. The proposed IR-QLoRA mainly relies on two technologies derived from the perspective of unified information: (1) statistics-based Information Calibration Quantization allows the quantized parameters of LLM to retain original information accurately; (2) finetuning-based Information Elastic Connection makes LoRA utilizes elastic representation transformation with diverse information. Comprehensive experiments show that IR-QLoRA can significantly improve accuracy across LLaMA and LLaMA2 families under 2-4 bit-widths, e.g., 4- bit LLaMA-7B achieves 1.4% improvement on MMLU compared with the state-of-the-art methods. The significant performance gain requires only a tiny 0.31% additional time consumption, revealing the satisfactory efficiency of our IRQLoRA. We highlight that IR-QLoRA enjoys excellent versatility, compatible with various frameworks (e.g., NormalFloat and Integer quantization) and brings general accuracy gains. The code is available at https://github.com/htqin/ir-qlora.

翻译：大语言模型的LoRA微调量化已被广泛研究，旨在获得精确且紧凑的模型以部署于资源受限的硬件。然而现有方法导致量化后的LLM性能严重退化，甚至无法从LoRA微调中获益。本文提出新型IR-QLoRA，通过信息保留推动采用LoRA的量化LLM实现高精度。该IR-QLoRA主要依赖从统一信息视角衍生的两项技术：(1) 基于统计的信息校准量化，使LLM的量化参数精确保留原始信息；(2) 基于微调的信息弹性连接，使LoRA利用具有多样信息的弹性表示变换。大量实验表明，在LLaMA和LLaMA2系列的2-4比特位宽下，IR-QLoRA能显著提升精度，例如4比特LLaMA-7B在MMLU上相比最先进方法提升1.4%。显著的性能提升仅需额外0.31%的时间消耗，揭示了IR-QLoRA令人满意的效率。我们强调IR-QLoRA具有优异的通用性，兼容多种框架（如NormalFloat和整数量化）并带来普遍精度增益。代码开源在https://github.com/htqin/ir-qlora。