We introduce HallusionBench, a comprehensive benchmark designed for the evaluation of image-context reasoning. This benchmark presents significant challenges to advanced large visual-language models (LVLMs), such as GPT-4V(Vision), Gemini Pro Vision, Claude 3, and LLaVA-1.5, by emphasizing nuanced understanding and interpretation of visual data. The benchmark comprises 346 images paired with 1129 questions, all meticulously crafted by human experts. We introduce a novel structure for these visual questions designed to establish control groups. This structure enables us to conduct a quantitative analysis of the models' response tendencies, logical consistency, and various failure modes. In our evaluation on HallusionBench, we benchmarked 15 different models, highlighting a 31.42% question-pair accuracy achieved by the state-of-the-art GPT-4V. Notably, all other evaluated models achieve accuracy below 16%. Moreover, our analysis not only highlights the observed failure modes, including language hallucination and visual illusion, but also deepens an understanding of these pitfalls. Our comprehensive case studies within HallusionBench shed light on the challenges of hallucination and illusion in LVLMs. Based on these insights, we suggest potential pathways for their future improvement. The benchmark and codebase can be accessed at https://github.com/tianyi-lab/HallusionBench.

翻译：我们引入HallusionBench，这是一个专为图像上下文推理评估而设计的综合性基准测试集。该基准通过强调对视觉数据的细致理解与解读，对GPT-4V(Vision)、Gemini Pro Vision、Claude 3及LLaVA-1.5等先进大型视觉语言模型构成了显著挑战。该基准涵盖346幅图像及其对应的1129个问题，所有内容均由人类专家精心构建。我们为这些视觉问题引入了一种新型结构，旨在建立控制组。该结构使我们能够对模型的响应倾向、逻辑一致性以及各类故障模式进行定量分析。在针对HallusionBench的评估中，我们对15种不同模型进行了基准测试，结果显示最先进的GPT-4V在问题对层面达到31.42%的准确率。值得注意的是，其余所有被评估模型的准确率均低于16%。此外，我们的分析不仅凸显了包括语言幻觉和视觉错觉在内的观测故障模式，还加深了对这些陷阱的理解。HallusionBench中的综合案例研究揭示了LVLMs在幻觉与错觉方面的挑战。基于这些洞见，我们提出了其未来改进的潜在路径。该基准及代码库可通过https://github.com/tianyi-lab/HallusionBench访问。