Low-rank adaption (LoRA) is a representative method in the field of parameter-efficient fine-tuning (PEFT), and is key to Democratizating the modern large language models (LLMs). The vanilla LoRA is implemented with uniform ranks, and the recent literature have found that properly allocating ranks on the LLM backbones results in performance boosts. However, the previous rank allocation methods have limitations since they rely on inexplanable and unreliable importance measures for the LoRA ranks. To address the above issues, we propose the ShapLoRA framework. Inspired by the explanable attribution measure Shapley Value, we combine the sensitivity-based measures with the idea of coalitions in the collaborative games among LoRA ranks, and propose a more explainable importance measure called Shapley sensitivity. In addition, we optimize the workflow of the existing works by: (a) calculating Shapley sensitivity on a separate validation set; (b) Setting up the allocating-retraining procedures for fair comparisons. We have conducted experiments on various challenging tasks, and the experimental results demonstrate that our ShapLoRA method can outperform the recent baselines with comparable tunable parameters.\footnote{Codes and fine-tuned models will be open-sourced to facilitate future research.

翻译：低秩适配（LoRA）是参数高效微调（PEFT）领域的代表性方法，也是实现现代大语言模型（LLM）民主化的关键技术。原始LoRA采用统一秩进行实现，近期研究发现，在LLM骨干网络上合理分配秩能够带来性能提升。然而，现有秩分配方法存在局限性，因其依赖于难以解释且不可靠的LoRA秩重要性度量指标。为解决上述问题，我们提出ShapLoRA框架。受可解释归因度量指标Shapley值的启发，我们将基于敏感度的度量方法与LoRA秩协作博弈中的联盟思想相结合，提出了一种更具可解释性的重要性度量指标——Shapley敏感度。此外，我们通过以下方式优化现有工作流程：（a）在独立验证集上计算Shapley敏感度；（b）建立分配-重训练流程以确保公平比较。我们在多个具有挑战性的任务上进行了实验，结果表明，在可调参数量相当的情况下，我们的ShapLoRA方法能够超越现有基线方法。\footnote{代码与微调模型将开源以促进后续研究。