As one of the most popular parameter-efficient fine-tuning (PEFT) methods, low-rank adaptation (LoRA) is commonly applied to fine-tune large language models (LLMs). However, updating the weights of LoRA blocks effectively and expeditiously is challenging due to the long calculation path in the original model. To address this, we propose ResLoRA, an improved framework of LoRA. By adding residual paths during training and using merging approaches to eliminate these extra paths during inference, our method can achieve better results in fewer training steps without any extra trainable parameters or inference cost compared to LoRA. The experiments on NLG, NLU, and text-to-image tasks demonstrate the effectiveness of our method. To the best of our knowledge, ResLoRA is the first work that combines the residual path with LoRA. The code of our method is available at https://github.com/microsoft/LMOps/tree/main/reslora .

翻译：作为最流行的参数高效微调方法之一，低秩适配（LoRA）通常被应用于微调大语言模型。然而，由于原始模型中的长计算路径，有效且快速地更新LoRA模块的权重颇具挑战性。为此，我们提出ResLoRA——一种LoRA的改进框架。通过在训练过程中添加残差路径，并在推理阶段利用合并方法消除这些额外路径，我们的方法相比LoRA能在更少的训练步骤中取得更优结果，且无需额外可训练参数或推理开销。在自然语言生成、自然语言理解和文本到图像任务上的实验证明了该方法的有效性。据我们所知，ResLoRA是首个将残差路径与LoRA相结合的工作。我们的方法代码已开源至 https://github.com/microsoft/LMOps/tree/main/reslora。