We propose Tied-LoRA, a simple paradigm utilizes weight tying and selective training to further increase parameter efficiency of the Low-rank adaptation (LoRA) method. Our investigations include all feasible combinations parameter training/freezing in conjunction with weight tying to identify the optimal balance between performance and the number of trainable parameters. Through experiments covering a variety of tasks and two base language models, we provide analysis revealing trade-offs between efficiency and performance. Our experiments uncovered a particular Tied-LoRA configuration that stands out by demonstrating comparable performance across several tasks while employing only 13~\% percent of parameters utilized by the standard LoRA method.

翻译：我们提出Tied-LoRA，一种利用权重绑定与选择性训练来进一步提升低秩适配（LoRA）方法参数效率的简单范式。我们的研究涵盖了权重绑定与参数训练/冻结的所有可行组合，以识别性能与可训练参数数量之间的最优平衡。通过涵盖多种任务和两个基础语言模型的实验，我们提供了揭示效率与性能之间权衡的分析。实验发现了一种特定的Tied-LoRA配置，该配置在仅使用标准LoRA方法13%参数的情况下，在多项任务上展现出可比拟的性能。