Routine clinical visits of a patient produce not only image data, but also non-image data containing clinical information regarding the patient, i.e., medical data is multi-modal in nature. Such heterogeneous modalities offer different and complementary perspectives on the same patient, resulting in more accurate clinical decisions when they are properly combined. However, despite its significance, how to effectively fuse the multi-modal medical data into a unified framework has received relatively little attention. In this paper, we propose an effective graph-based framework called HetMed (Heterogeneous Graph Learning for Multi-modal Medical Data Analysis) for fusing the multi-modal medical data. Specifically, we construct a multiplex network that incorporates multiple types of non-image features of patients to capture the complex relationship between patients in a systematic way, which leads to more accurate clinical decisions. Extensive experiments on various real-world datasets demonstrate the superiority and practicality of HetMed. The source code for HetMed is available at https://github.com/Sein-Kim/Multimodal-Medical.

翻译：患者的常规临床就诊不仅产生影像数据，还包含患者临床信息的非影像数据，即医疗数据本质上是多模态的。这些异质模态为同一患者提供了不同且互补的视角，当它们被恰当融合时，能够实现更精确的临床决策。然而，尽管其重要性显著，如何将多模态医疗数据有效融合至统一框架的研究仍相对不足。本文提出了一种名为HetMed（面向多模态医疗数据分析的异构图学习）的基于图的有效框架，用于融合多模态医疗数据。具体而言，我们构建了一个多重网络，该网络整合了患者多种类型的非影像特征，以系统化方式捕捉患者间的复杂关系，从而达成更精准的临床决策。在多种真实世界数据集上的大量实验证明了HetMed的优越性与实用性。HetMed的源代码已公开于https://github.com/Sein-Kim/Multimodal-Medical。