Machine learning models built on training data with multiple modalities can reveal new insights that are not accessible through unimodal datasets. For example, cardiac magnetic resonance images (MRIs) and electrocardiograms (ECGs) are both known to capture useful information about subjects' cardiovascular health status. A multimodal machine learning model trained from large datasets can potentially predict the onset of heart-related diseases and provide novel medical insights about the cardiovascular system. Despite the potential benefits, it is difficult for medical experts to explore multimodal representation models without visual aids and to test the predictive performance of the models on various subpopulations. To address the challenges, we developed a visual analytics system called Latent Space Explorer. Latent Space Explorer provides interactive visualizations that enable users to explore the multimodal representation of subjects, define subgroups of interest, interactively decode data with different modalities with the selected subjects, and inspect the accuracy of the embedding in downstream prediction tasks. A user study was conducted with medical experts and their feedback provided useful insights into how Latent Space Explorer can help their analysis and possible new direction for further development in the medical domain.

翻译：基于多模态训练数据构建的机器学习模型，能够揭示单模态数据集无法发现的新洞察。例如，心脏磁共振影像和心电图均可有效捕捉受试者心血管健康状态相关信息。通过大规模数据集训练的多模态机器学习模型，有望预测心脏相关疾病的发病征兆，并提供关于心血管系统的医学新见解。尽管具有潜在优势，但医学专家在缺乏可视化辅助工具的情况下，难以探索多模态表征模型并评估其在不同亚群中的预测性能。为解决这一挑战，我们开发了名为"潜在空间探索器"的可视分析系统。该系统通过交互式可视化，支持用户探索受试者的多模态表征、定义感兴趣的亚群、基于选定受试者交互式解码不同模态数据，并检验嵌入表示在下游预测任务中的准确性。我们开展了包含医学专家的用户研究，其反馈揭示了该系统如何辅助医学分析，并为医疗领域的进一步发展指明了新方向。