Instruction finetuning on a variety of image-text instruction data is the key to obtaining a versatile Multimodal Large Language Model (MLLM), and different configurations of the instruction data can lead to finetuned models with different capabilities. However, we have discovered that data conflicts are inevitable when mixing instruction data from distinct domains, which can result in performance drops for tasks of a specific domain. To address this issue, we propose to apply a sparse mixture of LoRA experts for instruction finetuning MLLMs. Within the Transformer layers, we extend the popular Low-Rank Adaption (LoRA) method by creating a set of LoRA experts specifically for the MLP layer, and route each token to the top-1 expert based on a routing function, allowing adaptive choices for tokens from different domains. Since the LoRA experts are sparsely activated, the training and inference cost are kept roughly constant compared to the original LoRA method. By replacing the plain-LoRA finetuing of LLaVA-1.5, our final model is named LLaVA-MoLE. Extensive experiments proved that LLaVA-MoLE effectively mitigates the data conflict issue when mixing multiple distinct instruction datasets with various configurations, and achieves consistent performance gains over the strong plain-LoRA baselines. Most importantly, on the mixed datasets, LLaVA-MoLE can even outperform the plain-LoRA baseline trained with twice the samples.

翻译：针对多样化的图像-文本指令数据进行指令微调是获得通用多模态大语言模型的关键，而指令数据的不同配置会导致微调模型具备不同的能力。然而，我们发现混合来自不同领域的指令数据时数据冲突不可避免，这可能导致特定领域任务的性能下降。为解决这一问题，我们提出在指令微调多模态大语言模型中应用稀疏混合LoRA专家。在Transformer层中，我们通过为MLP层创建一组专门的LoRA专家来扩展流行的低秩适配（LoRA）方法，并基于路由函数将每个令牌路由至排名第一的专家，使来自不同领域的令牌能够自适应选择专家。由于LoRA专家以稀疏方式激活，相较于原始LoRA方法，训练和推理成本基本保持不变。通过替换LLaVA-1.5中普通LoRA微调方法，我们的最终模型命名为LLaVA-MoLE。大量实验证明，LLaVA-MoLE在混合多个具有不同配置的指令数据集时能有效缓解数据冲突问题，并在强基线普通LoRA方法基础上实现持续性能提升。最重要的是，在混合数据集上，LLaVA-MoLE甚至能优于使用两倍样本训练的普通LoRA基线模型。