Over the last years, advancements in deep learning models for computer vision have led to a dramatic improvement in their image classification accuracy. However, models with a higher accuracy in the task they were trained on do not necessarily develop better image representations that allow them to also perform better in other tasks they were not trained on. In order to investigate the representation learning capabilities of prominent high-performing computer vision models, we investigated how well they capture various indices of perceptual similarity from large-scale behavioral datasets. We find that higher image classification accuracy rates are not associated with a better performance on these datasets, and in fact we observe no improvement in performance since GoogLeNet (released 2015) and VGG-M (released 2014). We speculate that more accurate classification may result from hyper-engineering towards very fine-grained distinctions between highly similar classes, which does not incentivize the models to capture overall perceptual similarities.

翻译：近年来，计算机视觉深度学习模型的进步显著提升了其在图像分类任务中的准确率。然而，在训练任务上具有更高准确率的模型，未必能发展出更优的图像表征以在其他未训练任务中表现出色。为探究高性能计算机视觉模型的表征学习能力，我们研究了这些模型如何捕捉大规模行为数据集中的多种感知相似性指标。研究发现，更高的图像分类准确率与这些数据集上的表现并无关联，事实上，自GoogLeNet（2015年发布）和VGG-M（2014年发布）以来，模型性能未见提升。我们推测，更高精度的分类可能源于对高度相似类别间极细粒度区分的过度工程化设计，这种导向未能激励模型捕捉整体感知相似性。