We are witnessing a widespread adoption of artificial intelligence in healthcare. However, most of the advancements in deep learning in this area consider only unimodal data, neglecting other modalities. Their multimodal interpretation necessary for supporting diagnosis, prognosis and treatment decisions. In this work we present a deep architecture, which jointly learns modality reconstructions and sample classifications using tabular and imaging data. The explanation of the decision taken is computed by applying a latent shift that, simulates a counterfactual prediction revealing the features of each modality that contribute the most to the decision and a quantitative score indicating the modality importance. We validate our approach in the context of COVID-19 pandemic using the AIforCOVID dataset, which contains multimodal data for the early identification of patients at risk of severe outcome. The results show that the proposed method provides meaningful explanations without degrading the classification performance.

翻译：当前，人工智能在医疗健康领域正得到广泛应用。然而，该领域深度学习的大多数进展仅考虑单模态数据，忽略了其他模态信息。而支持诊断、预后和治疗决策需要对这些模态进行多模态综合解读。本研究提出一种深度架构，能够利用表格数据和影像数据联合学习模态重建与样本分类。我们通过施加潜在偏移来计算决策的解释：该偏移模拟反事实预测，从而揭示各模态中对决策贡献最大的特征，并给出指示模态重要性的量化评分。我们使用包含多模态数据的AIforCOVID数据集，在COVID-19大流行的背景下验证了所提方法，该数据集旨在早期识别具有重症风险的患者。结果表明，所提方法能在保持分类性能不下降的前提下，提供具有实际意义的解释。