Generalist foundation models (GFMs) are renowned for their exceptional capability and flexibility in effectively generalizing across diverse tasks and modalities. In the field of medicine, while GFMs exhibit superior generalizability based on their extensive intrinsic knowledge as well as proficiency in instruction following and in-context learning, specialist models excel in precision due to their domain knowledge. In this work, for the first time, we explore the synergy between the GFM and specialist models, to enable precise medical image analysis on a broader scope. Specifically, we propose a cooperative framework, Generalist-Specialist Collaboration (GSCo), which consists of two stages, namely the construction of GFM and specialists, and collaborative inference on downstream tasks. In the construction stage, we develop MedDr, the largest open-source GFM tailored for medicine, showcasing exceptional instruction-following and in-context learning capabilities. Meanwhile, a series of lightweight specialists are crafted for downstream tasks with low computational cost. In the collaborative inference stage, we introduce two cooperative mechanisms, Mixture-of-Expert Diagnosis and Retrieval-Augmented Diagnosis, to harvest the generalist's in-context learning abilities alongside the specialists' domain expertise. For a comprehensive evaluation, we curate a large-scale benchmark featuring 28 datasets and about 250,000 images. Extensive results demonstrate that MedDr consistently outperforms state-of-the-art GFMs on downstream datasets. Furthermore, GSCo exceeds both GFMs and specialists across all out-of-domain disease diagnosis datasets. These findings indicate a significant paradigm shift in the application of GFMs, transitioning from separate models for specific tasks to a collaborative approach between GFMs and specialists, thereby advancing the frontiers of generalizable AI in medicine.

翻译：通才基础模型（GFMs）以其卓越的能力和灵活性，能够有效泛化至多种任务和模态而著称。在医学领域，尽管GFMs凭借其广泛的内在知识以及指令遵循和上下文学习能力展现出优异的泛化性，但专才模型因其领域知识而在精确性方面表现突出。在本工作中，我们首次探索了GFM与专才模型之间的协同作用，以实现在更广范围内进行精确的医学图像分析。具体而言，我们提出了一个协作框架——通才-专才协作（GSCo），该框架包含两个阶段：GFM与专才的构建阶段，以及下游任务的协同推理阶段。在构建阶段，我们开发了MedDr，这是目前最大的开源医学专用GFM，展现出卓越的指令遵循和上下文学习能力。同时，我们以较低的计算成本为下游任务构建了一系列轻量级专才模型。在协同推理阶段，我们引入了两种协作机制：专家混合诊断和检索增强诊断，以结合通才的上下文学习能力与专才的领域专业知识。为了进行全面评估，我们构建了一个包含28个数据集、约25万张图像的大规模基准。大量实验结果表明，MedDr在下游数据集上始终优于最先进的GFMs。此外，GSCo在所有域外疾病诊断数据集上的表现均超越了单独的GFMs和专才模型。这些发现标志着GFMs应用范式的重大转变，即从针对特定任务的独立模型转向GFMs与专才模型之间的协作，从而推动了医学领域通用人工智能的前沿发展。