Large Vision-Language Models (LVLMs), despite their recent success, are hardly comprehensively tested for their cognitive abilities. Inspired by the prevalent use of the "Cookie Theft" task in human cognition test, we propose a novel evaluation benchmark to evaluate high-level cognitive ability of LVLMs using images with rich semantics. It defines eight reasoning capabilities and consists of an image description task and a visual question answering task. Our evaluation on well-known LVLMs shows that there is still a large gap in cognitive ability between LVLMs and humans.

翻译：尽管大规模视觉语言模型（LVLMs）近期取得了显著成功，但其认知能力尚未得到全面测试。受人类认知测试中广泛使用的"饼干盗窃"任务启发，我们提出了一种新颖的评估基准，利用富含语义的图像来评估LVLMs的高阶认知能力。该基准定义了八项推理能力，包含图像描述任务和视觉问答任务。对知名LVLMs的评估结果表明，当前模型与人类认知能力之间仍存在显著差距。