Previous studies have compared neural activities in the visual cortex to representations in deep neural networks trained on image classification. Interestingly, while some suggest that their representations are highly similar, others argued the opposite. Here, we propose a new approach to characterize the similarity of the decision strategies of two observers (models or brains) using decision variable correlation (DVC). DVC quantifies the image-by-image correlation between the decoded decisions based on the internal neural representations in a classification task. Thus, it can capture task-relevant information rather than general representational alignment. We evaluate DVC using monkey V4/IT recordings and network models trained on image classification tasks. We find that model-model similarity is comparable to monkey-monkey similarity, whereas model-monkey similarity is consistently lower. Strikingly, DVC decreases with increasing network performance on ImageNet-1k. Adversarial training does not improve model-monkey similarity in task-relevant dimensions assessed using DVC, although it markedly increases the model-model similarity. Similarly, pre-training on larger datasets does not improve model-monkey similarity. These results suggest a divergence between the task-relevant representations in monkey V4/IT and those learned by models trained on image classification tasks.

翻译：先前的研究将视觉皮层中的神经活动与在图像分类任务上训练的深度神经网络表征进行了比较。有趣的是，尽管一些研究表明它们的表征高度相似，但其他研究却持相反观点。在此，我们提出一种新方法，利用决策变量相关性来刻画两个观察者（模型或大脑）决策策略的相似性。DVC量化了在分类任务中，基于内部神经表征解码出的决策在图像层面上的相关性。因此，它能够捕捉任务相关信息，而非一般的表征对齐。我们使用猴子的V4/IT区记录数据和在图像分类任务上训练的网络模型来评估DVC。我们发现，模型-模型相似性与猴子-猴子相似性相当，而模型-猴子相似性则持续较低。引人注目的是，DVC随着网络在ImageNet-1k数据集上性能的提升而降低。对抗训练虽然显著提高了模型-模型相似性，但并未改善使用DVC评估的任务相关维度上的模型-猴子相似性。同样，在更大数据集上进行预训练也未提高模型-猴子相似性。这些结果表明，猴子V4/IT区的任务相关表征与在图像分类任务上训练的模型所学到的表征之间存在差异。