Fusing structural-functional images of the brain has shown great potential to analyze the deterioration of Alzheimer's disease (AD). However, it is a big challenge to effectively fuse the correlated and complementary information from multimodal neuroimages. In this paper, a novel model termed cross-modal transformer generative adversarial network (CT-GAN) is proposed to effectively fuse the functional and structural information contained in functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI). The CT-GAN can learn topological features and generate multimodal connectivity from multimodal imaging data in an efficient end-to-end manner. Moreover, the swapping bi-attention mechanism is designed to gradually align common features and effectively enhance the complementary features between modalities. By analyzing the generated connectivity features, the proposed model can identify AD-related brain connections. Evaluations on the public ADNI dataset show that the proposed CT-GAN can dramatically improve prediction performance and detect AD-related brain regions effectively. The proposed model also provides new insights for detecting AD-related abnormal neural circuits.

翻译：融合大脑的结构与功能影像在分析阿尔茨海默病（Alzheimer’s disease, AD）恶化方面展现出巨大潜力。然而，如何有效融合多模态神经影像中相互关联且互补的信息仍是一大挑战。本文提出了一种名为跨模态Transformer生成对抗网络（CT-GAN）的新模型，旨在有效融合功能磁共振成像（fMRI）与弥散张量成像（DTI）所包含的功能与结构信息。CT-GAN能够以高效的端到端方式学习拓扑特征，并从多模态影像数据中生成多模态连接。此外，本文设计了交换双向注意力机制，逐步对齐共同特征并有效增强模态间的互补特征。通过分析所生成的连接特征，该模型可识别与AD相关的脑连接。在公开的ADNI数据集上的评估表明，所提出的CT-GAN能显著提升预测性能，并有效检测出与AD相关的脑区。该模型还为检测AD相关异常神经环路提供了新的见解。