Accurate predictive models of the visual cortex neural response to natural visual stimuli remain a challenge in computational neuroscience. In this work, we introduce V1T, a novel Vision Transformer based architecture that learns a shared visual and behavioral representation across animals. We evaluate our model on two large datasets recorded from mouse primary visual cortex and outperform previous convolution-based models by more than 12.7% in prediction performance. Moreover, we show that the self-attention weights learned by the Transformer correlate with the population receptive fields. Our model thus sets a new benchmark for neural response prediction and can be used jointly with behavioral and neural recordings to reveal meaningful characteristic features of the visual cortex.

翻译：精确预测视觉皮层对自然视觉刺激的神经反应模型仍是计算神经科学中的一项挑战。本文提出V1T——一种基于Vision Transformer的新型架构，能够学习跨动物的共享视觉与行为表征。我们在两个从小鼠初级视觉皮层记录的大型数据集上评估该模型，其预测性能比此前基于卷积的模型提升超过12.7%。此外，我们发现Transformer学习的自注意力权重与群体感受野相关。因此，本模型为神经反应预测设立了新基准，并可结合行为与神经记录揭示视觉皮层中有意义的特征。