Cognitive science research treats visual perception, the ability to understand and make sense of a visual input, as one of the early developmental signs of intelligence. Its TVPS-4 framework categorizes and tests human perception into seven skills such as visual discrimination, and form constancy. Do Multimodal Large Language Models (MLLMs) match up to humans in basic perception? Even though there are many benchmarks that evaluate MLLMs on advanced reasoning and knowledge skills, there is limited research that focuses evaluation on simple perception. In response, we introduce Percept-V, a dataset containing 6000 program-generated uncontaminated images divided into 30 domains, where each domain tests one or more TVPS-4 skills. Our focus is on perception, so we make our domains quite simple and the reasoning and knowledge required for solving them are minimal. Since modern-day MLLMs can solve much more complex tasks, our a-priori expectation is that they will solve these domains very easily. Contrary to our belief, our experiments show a weak performance of SoTA proprietary and open-source MLLMs compared to very high human performance on Percept-V. We find that as number of objects in the image increases, performance goes down rather fast. Our experiments also identify the perception skills that are considerably harder for all models.

翻译：认知科学研究将视觉感知——即理解并解读视觉输入的能力——视为智能早期发展的标志之一。其TVPS-4框架将人类感知能力分类并测试为七项技能，包括视觉辨别和形状恒常性等。多模态大语言模型（MLLMs）在基础感知能力上能否与人类相匹敌？尽管现有众多基准测试评估MLLMs在高级推理与知识技能上的表现，但专注于简单感知评估的研究却十分有限。为此，我们提出Percept-V数据集，包含6000张程序生成的未污染图像，划分为30个领域，每个领域测试一项或多项TVPS-4技能。由于研究聚焦于感知能力，我们设计的领域极为简单，所需推理与知识门槛极低。鉴于当代MLLMs能够解决更为复杂的任务，我们预期它们能轻松应对这些领域。然而与预期相反，实验表明最先进的专有及开源MLLMs在Percept-V上的表现远低于人类的高水平表现。我们发现随着图像中物体数量的增加，模型性能会快速下降。实验还识别出对所有模型都显著困难的感知技能类型。