Fine-tuning large-scale pre-trained models is inherently a resource-intensive task. While it can enhance the capabilities of the model, it also incurs substantial computational costs, posing challenges to the practical application of downstream tasks. Existing parameter-efficient fine-tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) rely on a bypass framework that ignores the differential parameter budget requirements across weight matrices, which may lead to suboptimal fine-tuning outcomes. To address this issue, we introduce the Dynamic Low-Rank Adaptation (DoRA) method. DoRA decomposes high-rank LoRA layers into structured single-rank components, allowing for dynamic pruning of parameter budget based on their importance to specific tasks during training, which makes the most of the limited parameter budget. Experimental results demonstrate that DoRA can achieve competitive performance compared with LoRA and full model fine-tuning, and outperform various strong baselines with the same storage parameter budget. Our code is available at https://github.com/MIkumikumi0116/DoRA

翻译：微调大规模预训练模型本质上是一项资源密集型任务。虽然它能增强模型能力，但也会产生大量计算成本，给下游任务的实际应用带来挑战。现有的参数高效微调方法（如低秩自适应）依赖于旁路框架，忽略了权重矩阵间差异化的参数预算需求，可能导致次优的微调结果。为解决此问题，我们提出了动态低秩自适应方法。DoRA将高秩LoRA层分解为结构化的单秩分量，允许根据训练过程中各分量对特定任务的重要性进行参数预算的动态剪枝，从而最大化有限参数预算的利用率。实验结果表明，DoRA在性能上可与LoRA及全模型微调相竞争，并在相同存储参数预算下优于多种强基线方法。我们的代码公开于https://github.com/MIkumikumi0116/DoRA。