We propose a novel VQA dataset, BloomVQA, to facilitate comprehensive evaluation of large vision-language models on comprehension tasks. Unlike current benchmarks that often focus on fact-based memorization and simple reasoning tasks without theoretical grounding, we collect multiple-choice samples based on picture stories that reflect different levels of comprehension, as laid out in Bloom's Taxonomy, a classic framework for learning assessment widely adopted in education research. Our data maps to a novel hierarchical graph representation which enables automatic data augmentation and novel measures characterizing model consistency. We perform graded evaluation and reliability analysis on recent multi-modal models. In comparison to low-level tasks, we observe decreased performance on tasks requiring advanced comprehension and cognitive skills with up to 38.0% drop in VQA accuracy. In comparison to earlier models, GPT-4V demonstrates improved accuracy over all comprehension levels while also shows a tendency of bypassing visual inputs especially for higher-level tasks. Current models also show consistency patterns misaligned with human comprehension in various scenarios, demonstrating the need of improvement based on theoretically-grounded criteria.

翻译：我们提出了一个新颖的VQA数据集——BloomVQA，旨在促进对大型视觉-语言模型在理解任务上的全面评估。与当前那些通常侧重于基于事实的记忆和简单推理任务、缺乏理论依据的基准不同，我们基于图画故事收集了多项选择样本，这些样本反映了布鲁姆分类学中描述的不同理解层次——布鲁姆分类学是教育研究中广泛采用的一种经典学习评估框架。我们的数据映射到一种新颖的层次化图表示，该表示支持自动数据增强并能够刻画模型一致性的新度量。我们对近期多模态模型进行了分级评估和可靠性分析。与低层次任务相比，我们观察到模型在需要高级理解与认知技能的任务上表现下降，VQA准确率降幅高达38.0%。与早期模型相比，GPT-4V在所有理解层次上的准确率均有提升，但也显示出尤其是在高层次任务中倾向于绕过视觉输入的趋势。当前模型在多种场景下还表现出与人类理解不一致的规律，凸显了基于理论依据的标准进行改进的必要性。