Fine-tuning large-scale pre-trained models is inherently a resource-intensive task. While it can enhance the capabilities of the model, it also incurs substantial computational costs, posing challenges to the practical application of downstream tasks. Existing parameter-efficient fine-tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) rely on a bypass framework that ignores the differential parameter budget requirements across weight matrices, which may lead to suboptimal fine-tuning outcomes. To address this issue, we introduce the Dynamic Low-Rank Adaptation (DoRA) method. DoRA decomposes high-rank LoRA layers into structured single-rank components, allowing for dynamic pruning of parameter budget based on their importance to specific tasks during training, which makes the most of the limited parameter budget. Experimental results demonstrate that DoRA can achieve competitive performance compared with LoRA and full model fine-tuning, and outperform various strong baselines with the same storage parameter budget. Our code is available at https://github.com/Yulongmao1/DoRA/

翻译：微调大规模预训练模型本质上是一项资源密集型任务。尽管它能提升模型能力，但也会产生巨大的计算成本，给下游任务的实际应用带来挑战。现有的参数高效微调（PEFT）方法，如低秩自适应（LoRA），依赖于旁路框架，忽略了不同权重矩阵对参数预算的差异化需求，这可能导致次优的微调结果。为解决这一问题，我们提出了动态低秩自适应（DoRA）方法。DoRA将高秩的LoRA层分解为结构化的单秩分量，允许在训练过程中根据其对特定任务的重要性动态剪裁参数预算，从而最大限度地利用有限的参数预算。实验结果表明，与LoRA和全模型微调相比，DoRA能够取得具有竞争力的性能，并且在相同的存储参数预算下优于多种强基线方法。我们的代码可在 https://github.com/Yulongmao1/DoRA/ 获取。