Deep learning approaches, together with neuroimaging techniques, play an important role in psychiatric disorders classification. Previous studies on psychiatric disorders diagnosis mainly focus on using functional connectivity matrices of resting-state functional magnetic resonance imaging (rs-fMRI) as input, which still needs to fully utilize the rich temporal information of the time series of rs-fMRI data. In this work, we proposed a multi-dimension-embedding-aware modality fusion transformer (MFFormer) for schizophrenia and bipolar disorder classification using rs-fMRI and T1 weighted structural MRI (T1w sMRI). Concretely, to fully utilize the temporal information of rs-fMRI and spatial information of sMRI, we constructed a deep learning architecture that takes as input 2D time series of rs-fMRI and 3D volumes T1w. Furthermore, to promote intra-modality attention and information fusion across different modalities, a fusion transformer module (FTM) is designed through extensive self-attention of hybrid feature maps of multi-modality. In addition, a dimension-up and dimension-down strategy is suggested to properly align feature maps of multi-dimensional from different modalities. Experimental results on our private and public OpenfMRI datasets show that our proposed MFFormer performs better than that using a single modality or multi-modality MRI on schizophrenia and bipolar disorder diagnosis.

翻译：深度学习方法与神经影像技术相结合，在精神疾病分类中发挥着重要作用。以往关于精神疾病诊断的研究主要依赖于使用静息态功能磁共振成像（rs-fMRI）的功能连接矩阵作为输入，这仍未能充分利用rs-fMRI数据时间序列中丰富的时序信息。本文提出了一种面向多维嵌入感知的模态融合Transformer（MFFormer），用于基于rs-fMRI和T1加权结构磁共振成像（T1w sMRI）的精神分裂症与双相情感障碍分类。具体而言，为充分利用rs-fMRI的时序信息与sMRI的空间信息，我们构建了一种深度学习架构，该架构以rs-fMRI的二维时间序列和T1w的三维体数据作为输入。此外，为促进模态内注意力机制及跨模态信息融合，通过多模态混合特征图的广泛自注意力机制设计了融合Transformer模块（FTM）。同时，提出了维度升维与降维策略，以恰当对齐来自不同模态的多维特征图。在私有数据集和公开OpenfMRI数据集上的实验结果表明，本文提出的MFFormer在精神分裂症与双相情感障碍诊断中优于仅使用单模态或多模态MRI的方法。