We propose a novel VQA dataset, BloomVQA, to facilitate comprehensive evaluation of large vision-language models on comprehension tasks. Unlike current benchmarks that often focus on fact-based memorization and simple reasoning tasks without theoretical grounding, we collect multiple-choice samples based on picture stories that reflect different levels of comprehension, as laid out in Bloom's Taxonomy, a classic framework for learning assessment widely adopted in education research. Our data maps to a novel hierarchical graph representation which enables automatic data augmentation and novel measures characterizing model consistency. We perform graded evaluation and reliability analysis on recent multi-modal models. In comparison to low-level tasks, we observe decreased performance on tasks requiring advanced comprehension and cognitive skills with up to 38.0\% drop in VQA accuracy. In comparison to earlier models, GPT-4V demonstrates improved accuracy over all comprehension levels and shows a tendency of bypassing visual inputs especially for higher-level tasks. Current models also show consistency patterns misaligned with human comprehension in various scenarios, demonstrating the need for improvement based on theoretically-grounded criteria.

翻译：我们提出了一种新颖的视觉问答数据集BloomVQA，旨在促进对大型视觉语言模型在理解任务上的全面评估。与当前通常侧重于基于事实的记忆和缺乏理论基础的简单推理任务的基准不同，我们基于图画故事收集了反映不同理解层次的多选题样本，这些层次依据教育研究中广泛采用的经典学习评估框架——布鲁姆分类法进行划分。我们的数据映射到一种新颖的层次化图表示，该表示支持自动数据增强和表征模型一致性的新型度量指标。我们对近期的多模态模型进行了分级评估和可靠性分析。与低层次任务相比，我们观察到模型在需要高级理解和认知技能的任务上性能下降，视觉问答准确率降幅最高达38.0%。与早期模型相比，GPT-4V在所有理解层次上均表现出更高的准确率，并且显示出绕过视觉输入的倾向，在高级任务中尤为明显。当前模型在各种场景下还表现出与人类理解不一致的模式，这表明需要基于理论依据的标准进行改进。