With the emergence of multimodal electronic health records, the evidence for an outcome may be captured across multiple modalities ranging from clinical to imaging and genomic data. Predicting outcomes effectively requires fusion frameworks capable of modeling fine-grained and multi-faceted complex interactions between modality features within and across patients. We develop an innovative fusion approach called MaxCorr MGNN that models non-linear modality correlations within and across patients through Hirschfeld-Gebelein-Renyi maximal correlation (MaxCorr) embeddings, resulting in a multi-layered graph that preserves the identities of the modalities and patients. We then design, for the first time, a generalized multi-layered graph neural network (MGNN) for task-informed reasoning in multi-layered graphs, that learns the parameters defining patient-modality graph connectivity and message passing in an end-to-end fashion. We evaluate our model an outcome prediction task on a Tuberculosis (TB) dataset consistently outperforming several state-of-the-art neural, graph-based and traditional fusion techniques.

翻译：摘要：随着多模态电子健康记录的出现，临床、影像及基因组数据等多种模态均可捕获与结局相关的证据。有效预测结局需要能够建模患者内及患者间模态特征之间细粒度、多层面复杂交互的融合框架。我们提出一种名为MaxCorrMGNN的创新融合方法，该方法通过赫希菲尔德-格贝莱因-伦伊最大相关（MaxCorr）嵌入来建模患者内及患者间的非线性模态相关性，生成保留模态与患者身份的多层图结构。在此基础上，我们首次设计了一种面向多层级图中任务感知推理的广义多层图神经网络（MGNN），该网络以端到端方式学习定义患者-模态图连通性与消息传递的参数。我们在结核病数据集上的结局预测任务中评估了该模型，其性能始终优于多种最先进的神经网络、基于图的传统融合技术。