Fusing structural-functional images of the brain has shown great potential to analyze the deterioration of Alzheimer's disease (AD). However, it is a big challenge to effectively fuse the correlated and complementary information from multimodal neuroimages. In this paper, a novel model termed cross-modal transformer generative adversarial network (CT-GAN) is proposed to effectively fuse the functional and structural information contained in functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI). The CT-GAN can learn topological features and generate multimodal connectivity from multimodal imaging data in an efficient end-to-end manner. Moreover, the swapping bi-attention mechanism is designed to gradually align common features and effectively enhance the complementary features between modalities. By analyzing the generated connectivity features, the proposed model can identify AD-related brain connections. Evaluations on the public ADNI dataset show that the proposed CT-GAN can dramatically improve prediction performance and detect AD-related brain regions effectively. The proposed model also provides new insights for detecting AD-related abnormal neural circuits.

翻译：融合大脑的结构与功能影像在分析阿尔茨海默病（AD）的恶化过程中展现出巨大潜力。然而，有效融合多模态神经影像中的相关性与互补性信息仍是一项重大挑战。本文提出一种名为跨模态Transformer生成对抗网络（CT-GAN）的新型模型，旨在高效融合功能磁共振成像（fMRI）与弥散张量成像（DTI）中包含的功能与结构信息。CT-GAN能够以端到端的高效方式从多模态成像数据中学习拓扑特征并生成多模态连接性。此外，设计了交换双注意力机制，逐步对齐模态间的共同特征，并有效增强互补特征。通过分析生成的连接性特征，所提模型可识别与AD相关的脑连接。在公开ADNI数据集上的评估表明，CT-GAN能够显著提升预测性能，并有效检测AD相关脑区。该模型还为检测AD相关的异常神经环路提供了新视角。