Investigating the mapping between visual stimuli and neural responses in the visual cortex contributes to a deeper understanding of biological visual processing mechanisms. Most existing studies characterize this mapping by training models to directly encode visual stimuli into neural responses or decode neural responses into visual stimuli. However, due to neural response variability and limited neural recording techniques, these studies suffer from overfitting and lack generalizability. Motivated by this challenge, in this paper we shift the tasks from conventional direct encoding and decoding to discriminative encoding and decoding, which are more reasonable. And on top of this we propose a cross-modal alignment approach, named Visual-Neural Alignment (VNA). To thoroughly test the performance of the three methods (direct encoding, direct decoding, and our proposed VNA) on discriminative encoding and decoding tasks, we conduct extensive experiments on three invasive visual cortex datasets, involving two types of subject mammals (mice and macaques). The results demonstrate that our VNA generally outperforms direct encoding and direct decoding, indicating our VNA can most precisely characterize the above visual-neural mapping among the three methods.

翻译：研究视觉刺激与视觉皮层神经响应之间的映射关系有助于深入理解生物视觉处理机制。现有研究大多通过训练模型将视觉刺激直接编码为神经响应或将神经响应解码为视觉刺激来表征这一映射。然而，由于神经响应的变异性及神经记录技术的局限性，这些研究存在过拟合问题且泛化能力不足。针对这一挑战，本文将任务从传统的直接编码与解码转向更具合理性的判别式编码与解码，并在此基础上提出一种跨模态对齐方法，称为视觉-神经对齐（VNA）。为全面测试三种方法（直接编码、直接解码及我们提出的VNA）在判别式编码与解码任务上的性能，我们在三个侵入式视觉皮层数据集上开展了大量实验，涉及两类哺乳动物被试（小鼠与猕猴）。结果表明，我们的VNA方法整体优于直接编码与直接解码，证明在三种方法中VNA能最精确地表征上述视觉-神经映射关系。