Face plays an important role in human's visual perception, and reconstructing perceived faces from brain activities is challenging because of its difficulty in extracting high-level features and maintaining consistency of multiple face attributes, such as expression, identity, gender, etc. In this study, we proposed a novel reconstruction framework, which we called Double-Flow GAN, that can enhance the capability of discriminator and handle imbalances in images from certain domains that are too easy for generators. We also designed a pretraining process that uses features extracted from images as conditions for making it possible to pretrain the conditional reconstruction model from fMRI in a larger pure image dataset. Moreover, we developed a simple pretrained model to perform fMRI alignment to alleviate the problem of cross-subject reconstruction due to the variations of brain structure among different subjects. We conducted experiments by using our proposed method and state-of-the-art reconstruction models. Our results demonstrated that our method showed significant reconstruction performance, outperformed the previous reconstruction models, and exhibited a good generation ability.

翻译：人脸在人类视觉感知中扮演着重要角色，从大脑活动中重建感知人脸因难以提取高层特征并保持表情、身份、性别等多重面部属性的一致性而极具挑战性。本研究提出了一种名为双流生成对抗网络的新型重建框架，该框架可增强判别器的能力，并处理生成器在特定领域图像中因过于简单而产生的数据不平衡问题。我们设计了一种预训练过程，利用从图像中提取的特征作为条件，使得在更大规模纯图像数据集上预训练基于功能磁共振成像的条件重建模型成为可能。此外，我们开发了一种简单的预训练模型来执行功能磁共振成像对齐，以缓解因不同受试者大脑结构差异导致的跨受试者重建问题。我们采用所提方法与当前最先进的重建模型进行了实验。结果表明，我们的方法展现出显著的重建性能，超越了以往的重建模型，并表现出良好的生成能力。