Over the past decade, predictive modeling of neural responses in the primate visual system has advanced significantly, largely driven by various DNN approaches. These include models optimized directly for visual recognition, cross-modal alignment through contrastive objectives, neural response prediction from scratch, and large language model embeddings.Likewise, different readout mechanisms, ranging from fully linear to spatial-feature factorized methods have been explored for mapping network activations to neural responses. Despite the diversity of these approaches, it remains unclear which method performs best across different visual regions. In this study, we systematically compare these approaches for modeling the human visual system and investigate alternative strategies to improve response predictions. Our findings reveal that for early to mid-level visual areas, response-optimized models with visual inputs offer superior prediction accuracy, while for higher visual regions, embeddings from LLMs based on detailed contextual descriptions of images and task-optimized models pretrained on large vision datasets provide the best fit. Through comparative analysis of these modeling approaches, we identified three distinct regions in the visual cortex: one sensitive primarily to perceptual features of the input that are not captured by linguistic descriptions, another attuned to fine-grained visual details representing semantic information, and a third responsive to abstract, global meanings aligned with linguistic content. We also highlight the critical role of readout mechanisms, proposing a novel scheme that modulates receptive fields and feature maps based on semantic content, resulting in an accuracy boost of 3-23% over existing SOTAs for all models and brain regions. Together, these findings offer key insights into building more precise models of the visual system.

翻译：过去十年间，灵长类视觉系统神经响应的预测建模取得了显著进展，这主要得益于各种深度神经网络方法的推动。这些方法包括直接针对视觉识别进行优化的模型、通过对比目标实现跨模态对齐的模型、从零开始预测神经响应的模型，以及基于大语言模型嵌入的方法。与此同时，从完全线性到空间特征分解等不同的读出机制也被用于将网络激活映射到神经响应。尽管方法多样，但目前尚不清楚哪种方法在不同视觉脑区中表现最优。本研究系统比较了这些用于人类视觉系统建模的方法，并探索了改进响应预测的替代策略。我们的研究结果表明：对于早期至中级视觉区域，采用视觉输入的响应优化模型能提供更优的预测精度；而对于高级视觉区域，基于图像详细上下文描述的大语言模型嵌入与在大规模视觉数据集上预训练的任务优化模型则表现出最佳拟合效果。通过对这些建模方法的比较分析，我们在视觉皮层中识别出三个功能特异的区域：一个主要对输入中未被语言描述捕获的感知特征敏感，另一个专注于表征语义信息的细粒度视觉细节，第三个则响应与语言内容对齐的抽象全局意义。我们还强调了读出机制的关键作用，提出了一种基于语义内容调节感受野与特征图的新方案，该方案使所有模型及脑区的预测精度较现有最优方法提升了3-23%。这些发现共同为构建更精确的视觉系统模型提供了关键见解。