Machine learning methods in healthcare have traditionally focused on using data from a single modality, limiting their ability to effectively replicate the clinical practice of integrating multiple sources of information for improved decision making. Clinicians typically rely on a variety of data sources including patients' demographic information, laboratory data, vital signs and various imaging data modalities to make informed decisions and contextualise their findings. Recent advances in machine learning have facilitated the more efficient incorporation of multimodal data, resulting in applications that better represent the clinician's approach. Here, we provide a review of multimodal machine learning approaches in healthcare, offering a comprehensive overview of recent literature. We discuss the various data modalities used in clinical diagnosis, with a particular emphasis on imaging data. We evaluate fusion techniques, explore existing multimodal datasets and examine common training strategies.

翻译：医疗健康领域的机器学习方法传统上侧重于使用单一模态的数据，这限制了其有效复制临床实践中整合多源信息以优化决策的能力。临床医师通常依赖多种数据来源，包括患者人口统计学信息、实验室数据、生命体征以及多种影像数据模态，以做出明智决策并为其发现提供背景支持。近期机器学习的进步促进了多模态数据更高效的整合，催生了更能体现临床医师工作方法的实际应用。本文对医疗健康领域的多模态机器学习方法进行了综述，全面梳理了近期文献。我们讨论了临床诊断中使用的各类数据模态，特别关注影像数据。我们评估了融合技术，探索了现有的多模态数据集，并审视了常见的训练策略。