One of the most impactful findings in computational neuroscience over the past decade is that the object recognition accuracy of deep neural networks (DNNs) correlates with their ability to predict neural responses to natural images in the inferotemporal (IT) cortex. This discovery supported the long-held theory that object recognition is a core objective of the visual cortex, and suggested that more accurate DNNs would serve as better models of IT neuron responses to images. Since then, deep learning has undergone a revolution of scale: billion parameter-scale DNNs trained on billions of images are rivaling or outperforming humans at visual tasks including object recognition. Have today's DNNs become more accurate at predicting IT neuron responses to images as they have grown more accurate at object recognition? Surprisingly, across three independent experiments, we find this is not the case. DNNs have become progressively worse models of IT as their accuracy has increased on ImageNet. To understand why DNNs experience this trade-off and evaluate if they are still an appropriate paradigm for modeling the visual system, we turn to recordings of IT that capture spatially resolved maps of neuronal activity elicited by natural images. These neuronal activity maps reveal that DNNs trained on ImageNet learn to rely on different visual features than those encoded by IT and that this problem worsens as their accuracy increases. We successfully resolved this issue with the neural harmonizer, a plug-and-play training routine for DNNs that aligns their learned representations with humans. Our results suggest that harmonized DNNs break the trade-off between ImageNet accuracy and neural prediction accuracy that assails current DNNs and offer a path to more accurate models of biological vision.

翻译：过去十年计算神经科学领域最具影响力的发现之一是：深度神经网络（DNN）的目标识别准确率与其预测颞下（IT）皮层对自然图像神经反应的能力相关。这一发现支持了长期以来的理论——目标识别是视觉皮层的核心目标，并表明更准确的DNN将成为更好的IT神经元对图像反应模型。自那以后，深度学习经历了一场规模革命：在数十亿图像上训练的数十亿参数级DNN在视觉任务（包括目标识别）中正与人类媲美甚至超越人类。随着DNN在目标识别准确性上的提升，它们对IT神经元图像反应预测的准确性是否也随之提高？令人惊讶的是，通过三项独立实验，我们发现事实并非如此。随着DNN在ImageNet上的准确性提高，它们已逐渐成为越来越差的IT模型。为了解DNN为何经历这种权衡并评估其是否仍是建模视觉系统的合适范式，我们转向了能够捕捉自然图像诱发的神经元活动空间解析图的IT记录。这些神经元活动图表明，在ImageNet上训练的DNN学习依赖的视觉特征与IT编码的特征不同，且这一问题随准确性提高而加剧。我们通过神经协调器（neural harmonizer）——一种即插即用的DNN训练程序，可将其学习表征与人类对齐——成功解决了这一问题。我们的结果表明，协调后的DNN打破了当前DNN所面临的ImageNet准确率与神经预测准确率之间的权衡，并为构建更准确的生物视觉模型提供了路径。