Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA have significantly improved the adaptation of LLMs to downstream tasks in a resource-efficient manner. However, in multi-task scenarios, challenges such as training imbalance and the seesaw effect frequently emerge. Mixture-of-LoRA (MoLoRA), which combines LoRA with sparse Mixture-of-Experts, mitigates some of these issues by promoting task-specific learning across experts. Despite this, MoLoRA remains inefficient in terms of training speed, parameter utilization, and overall multi-task performance. In this paper, we propose Mixture of Asymmetric Low-Rank Adaptaion (MALoRA), a flexible fine-tuning framework that leverages asymmetric optimization across LoRA experts. MALoRA reduces the number of trainable parameters by 30% to 48%, increases training speed by 1.2x, and matches the computational efficiency of single-task LoRA models. Additionally, MALoRA addresses overfitting issues commonly seen in high-rank configurations, enhancing performance stability. Extensive experiments across diverse multi-task learning scenarios demonstrate that MALoRA consistently outperforms all baseline methods in both inter-domain and intra-domain tasks.

翻译：参数高效微调（PEFT）方法（如LoRA）以资源高效的方式显著提升了大型语言模型在下游任务上的适应能力。然而，在多任务场景中，训练不平衡和跷跷板效应等挑战频繁出现。混合LoRA（MoLoRA）通过将LoRA与稀疏混合专家模型相结合，促进了专家间的任务特定学习，从而缓解了部分问题。尽管如此，MoLoRA在训练速度、参数利用率和整体多任务性能方面仍存在效率不足。本文提出非对称低秩适配混合（MALoRA），一种灵活的微调框架，利用LoRA专家间的非对称优化。MALoRA将可训练参数数量减少了30%至48%，训练速度提升了1.2倍，并达到了单任务LoRA模型的计算效率。此外，MALoRA解决了高秩配置中常见的过拟合问题，增强了性能稳定性。在多任务学习场景下的广泛实验表明，MALoRA在跨领域和领域内任务中均持续优于所有基线方法。