The advent of large language models (LLMs) has revolutionized natural language processing, enabling unprecedented capabilities in understanding and generating human-like text. However, the computational cost and convergence times associated with fine-tuning these models remain significant challenges. Low-Rank Adaptation (LoRA) has emerged as a promising method to mitigate these issues by introducing efficient fine-tuning techniques with a reduced number of trainable parameters. In this paper, we present OLoRA, an enhancement to the LoRA method that leverages orthonormal matrix initialization through QR decomposition. OLoRA significantly accelerates the convergence of LLM training while preserving the efficiency benefits of LoRA, such as the number of trainable parameters and GPU memory footprint. Our empirical evaluations demonstrate that OLoRA not only converges faster but also exhibits improved performance compared to standard LoRA across a variety of language modeling tasks. This advancement opens new avenues for more efficient and accessible fine-tuning of LLMs, potentially enabling broader adoption and innovation in natural language applications.

翻译：大语言模型（LLM）的出现彻底改变了自然语言处理领域，使其在理解和生成类人文本方面具备了前所未有的能力。然而，与微调这些模型相关的计算成本和收敛时间仍然是重大挑战。低秩适应（LoRA）作为一种有前景的方法应运而生，它通过引入可训练参数数量减少的高效微调技术来缓解这些问题。本文提出OLoRA，这是对LoRA方法的一种改进，它利用QR分解实现正交矩阵初始化。OLoRA在保持LoRA效率优势（如可训练参数数量和GPU内存占用）的同时，显著加速了LLM训练的收敛速度。我们的实证评估表明，与标准LoRA相比，OLoRA不仅在多种语言建模任务上收敛更快，而且表现出更优的性能。这一进展为更高效、更易实现的大语言模型微调开辟了新途径，有望推动自然语言应用得到更广泛的采纳和创新。