The rapid development of parameter-efficient fine-tuning methods has noticeably improved the efficiency of adapting large language models. Among these, LoRA has gained widespread popularity due to its strong balance of effectiveness and parameter efficiency. However, LoRA relies on initializing two low-rank matrices whose product is zero, which limits its ability to effectively activate and leverage the original model weights-creating a potential bottleneck for optimal performance. To address this limitation, we propose \textbf{IniLoRA}, a novel initialization strategy that initializes the low-rank matrices to closely approximate the original model weights. Experimental results indicate that IniLoRA achieves better performance than LoRA across a range of models and tasks. Additionally, we introduce two variants, IniLoRA-$α$ and IniLoRA-$β$, both leveraging distinct initialization methods to enhance performance further.

翻译：参数高效微调方法的快速发展显著提升了大型语言模型适配的效率。其中，LoRA因其在效果与参数效率间的良好平衡而广受欢迎。然而，LoRA依赖于初始化两个乘积为零的低秩矩阵，这限制了其有效激活和利用原始模型权重的能力，从而可能成为性能最优化的瓶颈。为解决这一局限，我们提出了\textbf{IniLoRA}，一种新颖的初始化策略，该策略将低秩矩阵初始化为紧密逼近原始模型权重的状态。实验结果表明，IniLoRA在多种模型和任务上均取得了优于LoRA的性能。此外，我们引入了两个变体——IniLoRA-$α$与IniLoRA-$β$，二者均利用不同的初始化方法以进一步提升性能。