Deep learning methods have advanced quickly in brain imaging analysis over the past few years, but they are usually restricted by the limited labeled data. Pre-trained model on unlabeled data has presented promising improvement in feature learning in many domains, including natural language processing and computer vision. However, this technique is under-explored in brain network analysis. In this paper, we focused on pre-training methods with Transformer networks to leverage existing unlabeled data for brain functional network classification. First, we proposed a Transformer-based neural network, named as BrainNPT, for brain functional network classification. The proposed method leveraged <cls> token as a classification embedding vector for the Transformer model to effectively capture the representation of brain network. Second, We proposed a pre-training architecture with two pre-training strategies for BrainNPT model to leverage unlabeled brain network data to learn the structure information of brain networks. The results of classification experiments demonstrated the BrainNPT model without pre-training achieved the best performance with the state-of-the-art models, and the BrainNPT model with pre-training strongly outperformed the state-of-the-art models. The pre-training BrainNPT model improved 8.75% of accuracy compared with the model without pre-training. We further compared the pre-training strategies, analyzed the influence of the parameters of the model, and interpreted the fine-tuned model.

翻译：深度学习方法在过去几年中在脑成像分析领域取得了快速发展，但通常受限于有限的标注数据。在自然语言处理和计算机视觉等多个领域中，基于无标注数据的预训练模型已在特征学习方面展现出显著提升。然而，该技术在脑网络分析中尚未得到充分探索。本文聚焦于利用Transformer网络的预训练方法，以充分利用现有无标注数据进行脑功能网络分类。首先，我们提出了一种基于Transformer的神经网络——BrainNPT，用于脑功能网络分类。该方法利用<cls>标记作为Transformer模型的分类嵌入向量，有效捕获脑网络的表征。其次，我们为BrainNPT模型设计了一种包含两种预训练策略的预训练架构，以利用无标注脑网络数据学习脑网络的结构信息。分类实验结果表明，未经过预训练的BrainNPT模型达到了与最先进模型相当的性能，而经过预训练的BrainNPT模型则显著优于最先进模型。与未预训练模型相比，预训练BrainNPT模型的准确率提升了8.75%。我们进一步比较了预训练策略，分析了模型参数的影响，并对微调后的模型进行了解释。