Multi-modal large language models (MLLMs) have shown impressive capabilities as a general-purpose interface for various visual and linguistic tasks. However, building a unified MLLM for multi-task learning in the medical field remains a thorny challenge. To mitigate the tug-of-war problem of multi-modal multi-task optimization in MLLMs, recent advances primarily focus on improving the LLM components, while neglecting the connector that bridges the gap between modalities. In this paper, we introduce Uni-Med, a novel medical generalist foundation model which consists of a universal visual feature extraction module, a connector mixture-of-experts (CMoE) module, and an LLM. Benefiting from the proposed CMoE that leverages a well-designed router with a mixture of projection experts at the connector, Uni-Med achieves efficient solution to the tug-of-war problem and can perform six different medical tasks including question answering, visual question answering, report generation, referring expression comprehension, referring expression generation and image classification. To the best of our knowledge, Uni-Med is the first effort to tackle multi-task interference at the connector in MLLMs. Extensive ablation experiments validate the effectiveness of introducing CMoE under any configuration, with up to an average 8% performance gains. We further provide interpretation analysis of the tug-of-war problem from the perspective of gradient optimization and parameter statistics. Compared to previous state-of-the-art medical MLLMs, Uni-Med achieves competitive or superior evaluation metrics on diverse tasks. Code and resources are available at https://github.com/tsinghua-msiip/Uni-Med.

翻译：多模态大语言模型（MLLMs）作为处理各类视觉与语言任务的通用接口已展现出卓越能力。然而，在医学领域构建一个适用于多任务学习的统一MLLM仍面临严峻挑战。为缓解MLLM中多模态多任务优化的“拔河问题”，近期研究主要聚焦于改进大语言模型组件，却忽视了桥接模态差异的连接器模块。本文提出Uni-Med——一种新型医学通用基础模型，其核心架构包含通用视觉特征提取模块、连接器混合专家模块以及大语言模型。得益于所提出的连接器混合专家模块（CMoE），该模块通过精心设计的路由机制与混合投影专家网络，Uni-Med能有效解决多任务“拔河问题”，并可执行六类医学任务：问答、视觉问答、报告生成、指代表达理解、指代表达生成及图像分类。据我们所知，Uni-Med是首个针对MLLM连接器层面多任务干扰问题的系统性解决方案。大量消融实验证明，在任何配置下引入CMoE均能提升模型性能，平均增益最高达8%。我们进一步从梯度优化与参数统计的视角对“拔河问题”进行了解析分析。相较于现有最优医学MLLM，Uni-Med在多项任务中取得了具有竞争力或更优的评估指标。代码与资源已开源：https://github.com/tsinghua-msiip/Uni-Med。