Developing algorithms for accurate and comprehensive neural decoding of mental contents is one of the long-cherished goals in the field of neuroscience and brain-machine interfaces. Previous studies have demonstrated the feasibility of neural decoding by training machine learning models to map brain activity patterns into a semantic vector representation of stimuli. These vectors, hereafter referred as pretrained feature vectors, are usually derived from semantic spaces based solely on image and/or text features and therefore they might have a totally different characteristics than how visual stimuli is represented in the human brain, resulting in limiting the capability of brain decoders to learn this mapping. To address this issue, we propose a representation learning framework, termed brain-grounding of semantic vectors, which fine-tunes pretrained feature vectors to better align with the neural representation of visual stimuli in the human brain. We trained this model this model with functional magnetic resonance imaging (fMRI) of 150 different visual stimuli categories, and then performed zero-shot brain decoding and identification analyses on 1) fMRI and 2) magnetoencephalography (MEG). Interestingly, we observed that by using the brain-grounded vectors, the brain decoding and identification accuracy on brain data from different neuroimaging modalities increases. These findings underscore the potential of incorporating a richer array of brain-derived features to enhance performance of brain decoding algorithms.

翻译：开发用于准确且全面地神经解码心理内容的算法是神经科学与脑机接口领域长期追求的目标之一。先前研究已证明，通过训练机器学习模型将脑活动模式映射为刺激的语义向量表示，可实现神经解码的可行性。此类向量（以下称为预训练特征向量）通常基于仅依赖图像和/或文本特征的语义空间构建，因此可能与人类大脑对视觉刺激的表征特征存在根本差异，从而限制脑解码器学习这种映射的能力。为解决这一问题，我们提出了一种名为"语义向量的脑接地"的表征学习框架，该框架通过微调预训练特征向量，使其更紧密地对齐人类大脑中视觉刺激的神经表征。我们利用150种不同视觉刺激类别功能性磁共振成像（fMRI）数据训练该模型，并在1）fMRI和2）脑磁图（MEG）上执行零样本脑解码与脑识别分析。值得注意的是，我们发现使用脑接地向量后，对不同神经影像模态脑数据的解码与识别准确率均有提升。这些发现表明，整合更丰富的脑源性特征可增强脑解码算法的性能，具有重要潜力。