While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA have effectively addressed GPU memory constraints during fine-tuning, their performance often falls short, especially in multidimensional task scenarios. To address this issue, one straightforward solution is to introduce task-specific LoRA modules as domain experts, leveraging the modeling of multiple experts' capabilities and thus enhancing the general capability of multi-task learning. Despite promising, these additional components often add complexity to the training and inference process, contravening the efficient characterization of PEFT designed for. Considering this, we introduce an innovative PEFT method, TeamLoRA, consisting of a collaboration and competition module for experts, and thus achieving the right balance of effectiveness and efficiency: (i) For collaboration, a novel knowledge-sharing and -organizing mechanism is devised to appropriately reduce the scale of matrix operations, thereby boosting the training and inference speed. (ii) For competition, we propose leveraging a game-theoretic interaction mechanism for experts, encouraging experts to transfer their domain-specific knowledge while facing diverse downstream tasks, and thus enhancing the performance. By doing so, TeamLoRA elegantly connects the experts as a "Team" with internal collaboration and competition, enabling a faster and more accurate PEFT paradigm for multi-task learning. To validate the superiority of TeamLoRA, we curate a comprehensive multi-task evaluation(CME) benchmark to thoroughly assess the capability of multi-task learning. Experiments conducted on our CME and other benchmarks indicate the effectiveness and efficiency of TeamLoRA. Our project is available at https://github.com/Lin-Tianwei/TeamLoRA.

翻译：尽管LoRA等参数高效微调方法有效解决了微调过程中的GPU内存限制，但其性能表现往往不尽如人意，尤其是在多维任务场景中。为解决这一问题，一种直接的方案是引入任务特定的LoRA模块作为领域专家，利用对多专家能力的建模来提升多任务学习的通用能力。尽管前景可观，这些额外组件通常增加了训练和推理过程的复杂性，违背了参数高效微调方法所追求的高效特性。有鉴于此，我们提出了一种创新的参数高效微调方法TeamLoRA，该方法包含专家协作与竞争模块，从而在效果与效率之间实现了良好平衡：（i）在协作方面，设计了一种新颖的知识共享与组织机制，以适度降低矩阵运算规模，从而提升训练与推理速度。（ii）在竞争方面，我们提出利用博弈论交互机制激励专家，在面对多样化下游任务时促进专家传递其领域特定知识，进而提升性能。通过这种方式，TeamLoRA巧妙地将专家连接成一个具有内部协作与竞争关系的“团队”，为多任务学习实现了更快、更准确的参数高效微调范式。为验证TeamLoRA的优越性，我们构建了一个全面的多任务评估基准，以系统评估多任务学习能力。在我们构建的基准及其他基准上进行的实验均证明了TeamLoRA的有效性与高效性。项目开源地址：https://github.com/Lin-Tianwei/TeamLoRA。