Quantization is an indispensable technique for serving Large Language Models (LLMs) and has recently found its way into LoRA fine-tuning. In this work we focus on the scenario where quantization and LoRA fine-tuning are applied together on a pre-trained model. In such cases it is common to observe a consistent gap in the performance on downstream tasks between full fine-tuning and quantization plus LoRA fine-tuning approach. In response, we propose LoftQ (LoRA-Fine-Tuning-aware Quantization), a novel quantization framework that simultaneously quantizes an LLM and finds a proper low-rank initialization for LoRA fine-tuning. Such an initialization alleviates the discrepancy between the quantized and full-precision model and significantly improves generalization in downstream tasks. We evaluate our method on natural language understanding, question answering, summarization, and natural language generation tasks. Experiments show that our method is highly effective and outperforms existing quantization methods, especially in the challenging 2-bit and 2/4-bit mixed precision regimes. The code is available on https://github.com/yxli2123/LoftQ.

翻译：量化是服务于大语言模型（LLM）不可或缺的技术，并已应用于LoRA微调。本文聚焦于在预训练模型上同时应用量化与LoRA微调的场景。在此类情形下，相比于全模型微调，"量化+LoRA微调"方法在下游任务性能上通常存在显著差距。为此，我们提出LoftQ（LoRA微调感知量化），一种新型量化框架，该框架在量化大语言模型的同时，为LoRA微调寻找合适的低秩初始化。这种初始化方式可缩小量化模型与全精度模型之间的差异，并显著提升下游任务的泛化能力。我们在自然语言理解、问答、文本摘要及自然语言生成任务上评估了该方法。实验表明，我们的方法性能显著超越现有量化方法，尤其在具有挑战性的2比特及2/4比特混合精度场景下表现突出。代码已开源至https://github.com/yxli2123/LoftQ。