Brain decoding, a pivotal field in neuroscience, aims to reconstruct stimuli from acquired brain signals, primarily utilizing functional magnetic resonance imaging (fMRI). Currently, brain decoding is confined to a per-subject-per-model paradigm, limiting its applicability to the same individual for whom the decoding model is trained. This constraint stems from three key challenges: 1) the inherent variability in input dimensions across subjects due to differences in brain size; 2) the unique intrinsic neural patterns, influencing how different individuals perceive and process sensory information; 3) limited data availability for new subjects in real-world scenarios hampers the performance of decoding models. In this paper, we present a novel approach, MindBridge, that achieves cross-subject brain decoding by employing only one model. Our proposed framework establishes a generic paradigm capable of addressing these challenges by introducing biological-inspired aggregation function and novel cyclic fMRI reconstruction mechanism for subject-invariant representation learning. Notably, by cycle reconstruction of fMRI, MindBridge can enable novel fMRI synthesis, which also can serve as pseudo data augmentation. Within the framework, we also devise a novel reset-tuning method for adapting a pretrained model to a new subject. Experimental results demonstrate MindBridge's ability to reconstruct images for multiple subjects, which is competitive with dedicated subject-specific models. Furthermore, with limited data for a new subject, we achieve a high level of decoding accuracy, surpassing that of subject-specific models. This advancement in cross-subject brain decoding suggests promising directions for wider applications in neuroscience and indicates potential for more efficient utilization of limited fMRI data in real-world scenarios. Project page: https://littlepure2333.github.io/MindBridge

翻译：大脑解码是神经科学中的关键领域，旨在从采集到的脑信号中重建刺激信息，主要利用功能性磁共振成像（fMRI）。目前，大脑解码局限于“每一被试、每一模型”的范式，从而限制其仅能应用于训练解码模型的同一被试。这一约束源于三个关键挑战：1）由于大脑尺寸差异，不同被试的输入维度固有可变性；2）独特的固有神经模式，影响不同个体感知和处理感官信息的方式；3）真实场景中新被试的数据可用性有限，阻碍了解码模型的性能。本文提出一种新方法MindBridge，仅使用单一模型实现跨被试大脑解码。我们提出的框架建立了一个通用范式，能够通过引入生物学启发的聚合函数和新型循环fMRI重建机制进行被试不变表征学习，从而应对这些挑战。值得注意的是，通过fMRI的循环重建，MindBridge能够实现新型fMRI合成，该合成数据亦可作为伪数据增强。在该框架内，我们还设计了一种新的重置微调方法，用于使预训练模型适应新被试。实验结果表明，MindBridge能够为多个被试重建图像，其性能与专用的被试特定模型相当。此外，针对新被试，即使数据有限，我们也能达到超越被试特定模型的高水平解码精度。这一跨被试大脑解码的进展为神经科学领域的更广泛应用指明了有前景的方向，并表明在真实场景中更高效利用有限的fMRI数据具有潜力。项目页面：https://littlepure2333.github.io/MindBridge