Decoding human visual neural representations is a challenging task with great scientific significance in revealing vision-processing mechanisms and developing brain-like intelligent machines. Most existing methods are difficult to generalize to novel categories that have no corresponding neural data for training. The two main reasons are 1) the under-exploitation of the multimodal semantic knowledge underlying the neural data and 2) the small number of paired (stimuli-responses) training data. To overcome these limitations, this paper presents a generic neural decoding method called BraVL that uses multimodal learning of brain-visual-linguistic features. We focus on modeling the relationships between brain, visual and linguistic features via multimodal deep generative models. Specifically, we leverage the mixture-of-product-of-experts formulation to infer a latent code that enables a coherent joint generation of all three modalities. To learn a more consistent joint representation and improve the data efficiency in the case of limited brain activity data, we exploit both intra- and inter-modality mutual information maximization regularization terms. In particular, our BraVL model can be trained under various semi-supervised scenarios to incorporate the visual and textual features obtained from the extra categories. Finally, we construct three trimodal matching datasets, and the extensive experiments lead to some interesting conclusions and cognitive insights: 1) decoding novel visual categories from human brain activity is practically possible with good accuracy; 2) decoding models using the combination of visual and linguistic features perform much better than those using either of them alone; 3) visual perception may be accompanied by linguistic influences to represent the semantics of visual stimuli. Code and data: https://github.com/ChangdeDu/BraVL.

翻译：解码人类视觉神经表示是一项极具挑战性的任务，在揭示视觉处理机制和发展类脑智能机器方面具有重要科学意义。现有方法大多难以泛化到缺乏对应神经数据训练的新类别，主要原因有二：1）对神经数据蕴含的多模态语义知识开发不足；2）配对的（刺激-响应）训练数据量稀少。为克服这些局限，本文提出一种名为BraVL的通用神经解码方法，通过脑-视觉-语言特征的多模态学习实现。我们聚焦于利用多模态深度生成模型建模脑、视觉和语言特征之间的关系。具体而言，采用专家乘积混合公式推断潜在编码，实现三种模态的协同联合生成。为学习更一致的联合表征并提升有限脑活动数据情境下的数据效率，我们利用模态内和模态间互信息最大化正则化项。特别是，BraVL模型可在多种半监督场景下训练，融合来自额外类别的视觉和文本特征。最终，我们构建三个三模态匹配数据集，大量实验得出有趣结论与认知启示：1）从人类脑活动中解码新视觉类别在实践上可行且准确率较高；2）融合视觉与语言特征的解码模型性能显著优于单独使用其中任一种特征；3）视觉感知可能伴随语言影响以表征视觉刺激的语义。代码与数据：https://github.com/ChangdeDu/BraVL。