Routine clinical visits of a patient produce not only image data, but also non-image data containing clinical information regarding the patient, i.e., medical data is multi-modal in nature. Such heterogeneous modalities offer different and complementary perspectives on the same patient, resulting in more accurate clinical decisions when they are properly combined. However, despite its significance, how to effectively fuse the multi-modal medical data into a unified framework has received relatively little attention. In this paper, we propose an effective graph-based framework called HetMed (Heterogeneous Graph Learning for Multi-modal Medical Data Analysis) for fusing the multi-modal medical data. Specifically, we construct a multiplex network that incorporates multiple types of non-image features of patients to capture the complex relationship between patients in a systematic way, which leads to more accurate clinical decisions. Extensive experiments on various real-world datasets demonstrate the superiority and practicality of HetMed. The source code for HetMed is available at https://github.com/Sein-Kim/Multimodal-Medical.

翻译：病人的例行临床访问不仅产生图像数据,而且不产生含有病人临床信息的非图像数据,即医疗数据是多式的,这种多样性模式对同一病人提供不同和互补的观点,在适当结合时导致更准确的临床决定;然而,尽管其意义重大,如何将多式医疗数据有效地结合到一个统一的框架中却很少受到重视;在本文件中,我们提议了一个有效的图表框架,称为HetMed(多式医学数据分析的遗传图解学习),用于利用多式医疗数据。具体地说,我们建立了一个多式网络,其中包含病人的多种非图像特征,以系统的方式捕捉病人之间的复杂关系,从而导致更准确的临床决定。关于各种真实世界数据集的广泛实验显示了HetMed的优越性和实用性。HetMed的源代码见https://github.com/Sein-Kim/Multimdal-Medical。