Multi-modal representation methods have achieved advanced performance in medical applications by extracting more robust features from multi-domain data. However, existing methods usually need to train additional branches for downstream tasks, which may increase the model complexities in clinical applications as well as introduce additional human inductive bias. Besides, very few studies exploit the rich clinical knowledge embedded in clinical daily reports. To this end, we propose a novel medical generalist agent, MGA, that can address three kinds of common clinical tasks via clinical reports knowledge transformation. Unlike the existing methods, MGA can easily adapt to different tasks without specific downstream branches when their corresponding annotations are missing. More importantly, we are the first attempt to use medical professional language guidance as a transmission medium to guide the agent's behavior. The proposed method is implemented on four well-known X-ray open-source datasets, MIMIC-CXR, CheXpert, MIMIC-CXR-JPG, and MIMIC-CXR-MS. Promising results are obtained, which validate the effectiveness of our proposed MGA. Code is available at: https://github.com/SZUHvern/MGA

翻译：多模态表征方法通过从多领域数据中提取更具鲁棒性的特征，已在医学应用中取得显著性能。然而，现有方法通常需要为下游任务训练额外分支，这不仅增加了临床应用的模型复杂度，还可能引入人为归纳偏倚。此外，鲜有研究充分利用临床日常报告中蕴含的丰富临床知识。为此，我们提出一种新型医学通才智能体MGA，其通过临床报告知识转化可处理三类常见临床任务。与现有方法不同，MGA在缺乏相应标注时无需特定下游分支即可轻松适配不同任务。更重要的是，我们首次尝试将医学专业语言引导作为传输媒介来指导智能体行为。该方法在四个知名X光开源数据集（MIMIC-CXR、CheXpert、MIMIC-CXR-JPG和MIMIC-CXR-MS）上实现，取得的优异结果验证了所提MGA的有效性。代码开源地址：https://github.com/SZUHvern/MGA