In recent years, deep learning models have been applied to neuroimaging data for early diagnosis of Alzheimer's disease (AD). Structural magnetic resonance imaging (sMRI) and positron emission tomography (PET) images provide structural and functional information about the brain, respectively. Combining these features leads to improved performance than using a single modality alone in building predictive models for AD diagnosis. However, current multi-modal approaches in deep learning, based on sMRI and PET, are mostly limited to convolutional neural networks, which do not facilitate integration of both image and phenotypic information of subjects. We propose to use graph neural networks (GNN) that are designed to deal with problems in non-Euclidean domains. In this study, we demonstrate how brain networks can be created from sMRI or PET images and be used in a population graph framework that can combine phenotypic information with imaging features of these brain networks. Then, we present a multi-modal GNN framework where each modality has its own branch of GNN and a technique is proposed to combine the multi-modal data at both the level of node vectors and adjacency matrices. Finally, we perform late fusion to combine the preliminary decisions made in each branch and produce a final prediction. As multi-modality data becomes available, multi-source and multi-modal is the trend of AD diagnosis. We conducted explorative experiments based on multi-modal imaging data combined with non-imaging phenotypic information for AD diagnosis and analyzed the impact of phenotypic information on diagnostic performance. Results from experiments demonstrated that our proposed multi-modal approach improves performance for AD diagnosis, and this study also provides technical reference and support the need for multivariate multi-modal diagnosis methods.

翻译：近年来，深度学习模型被应用于神经影像数据，以实现阿尔茨海默病（AD）的早期诊断。结构性磁共振成像（sMRI）和正电子发射断层扫描（PET）图像分别提供大脑的结构和功能信息。与单独使用单一模态相比，结合这些特征可在构建AD诊断预测模型时提升性能。然而，当前基于sMRI和PET的深度学习方法大多局限于卷积神经网络，这类网络难以整合受试者的影像与表型信息。我们提出采用图神经网络（GNN），该网络专为处理非欧几里得域问题而设计。本研究展示了如何从sMRI或PET图像构建脑网络，并将其纳入群体图框架中，该框架能结合这些脑网络的影像特征与表型信息。随后，我们提出一种多模态GNN框架，其中每种模态拥有独立的GNN分支，并设计了一种技术在节点向量和邻接矩阵层面融合多模态数据。最终，我们通过后期融合合并各分支的初步决策，生成最终预测结果。随着多模态数据的普及，多源多模态已成为AD诊断的趋势。我们开展了基于多模态影像数据结合非影像表型信息的探索性实验，以分析表型信息对诊断性能的影响。实验结果表明，我们提出的多模态方法显著提升了AD诊断性能，同时本研究也为多变量多模态诊断方法的需求提供了技术参考与支持。