Multiple benchmarks have been developed to assess the alignment between deep neural networks (DNNs) and human vision. In almost all cases these benchmarks are observational in the sense they are composed of behavioural and brain responses to naturalistic images that have not been manipulated to test hypotheses regarding how DNNs or humans perceive and identify objects. Here we introduce the toolbox \textit{MindSet: Vision}, consisting of a collection of image datasets and related scripts designed to test DNNs on 30 psychological findings. In all experimental conditions, the stimuli are systematically manipulated to test specific hypotheses regarding human visual perception and object recognition. In addition to providing pre-generated datasets of images, we provide code to regenerate these datasets, offering many configurable parameters which greatly extend the dataset versatility for different research contexts, and code to facilitate the testing of DNNs on these image datasets using three different methods (similarity judgments, out-of-distribution classification, and decoder method), accessible via https://github.com/MindSetVision/MindSetVision. To illustrate the challenges these datasets pose for developing better DNN models of human vision, we test several models on range of datasets included in the toolbox.

翻译：为评估深度神经网络（DNN）与人类视觉的匹配程度，研究者们已开发出多种基准测试。几乎所有此类基准都基于观察性设计，即通过自然图像引发行为与大脑反应，但这些图像并未经系统操控以检验关于DNN或人类如何感知及识别对象的假设。本文介绍工具箱《MindSet: Vision》，其包含一组图像数据集及相关脚本，旨在30项心理学发现上对DNN进行测试。在所有实验条件下，刺激物均经过系统操控，以检验关于人类视觉感知与物体识别的特定假设。除提供预生成的图像数据集外，我们还提供了重新生成这些数据集的代码，其中包含众多可配置参数，极大增强了数据集在不同研究情境下的适用性；同时提供代码，支持通过三种不同方法（相似性判断、分布外分类和解码器方法）在这些图像数据集上测试DNN，用户可通过https://github.com/MindSetVision/MindSetVision访问。为阐明这些数据集对开发更优人类视觉DNN模型所构成的挑战，我们选取了若干模型，在工具箱包含的一系列数据集上进行了测试。