We introduce HallusionBench, a comprehensive benchmark designed for the evaluation of image-context reasoning. This benchmark presents significant challenges to advanced large visual-language models (LVLMs), such as GPT-4V(Vision), Gemini Pro Vision, and LLaVA-1.5, by emphasizing nuanced understanding and interpretation of visual data. The benchmark comprises 346 images paired with 1129 questions, all meticulously crafted by human experts. We introduce a novel structure for these visual questions designed to establish control groups. This structure enables us to conduct a quantitative analysis of the models' response tendencies, logical consistency, and various failure modes. In our evaluation on HallusionBench, we benchmarked 14 different models, highlighting a 31.42% question-pair accuracy achieved by the state-of-the-art GPT-4V. Notably, all other evaluated models achieve accuracy below 16%. Moreover, our analysis not only highlights the observed failure modes, including language hallucination and visual illusion, but also deepens an understanding of these pitfalls. Our comprehensive case studies within HallusionBench shed light on the challenges of hallucination and illusion in LVLMs. Based on these insights, we suggest potential pathways for their future improvement. The benchmark and codebase can be accessed at https://github.com/tianyi-lab/HallusionBench.

翻译：我们提出HallusionBench，一个专为评估图像上下文推理能力而设计的综合性基准测试。该基准通过强调对视觉数据的细微理解与解读，对GPT-4V(Vision)、Gemini Pro Vision及LLaVA-1.5等先进大视觉语言模型(LVLM)构成了重大挑战。基准包含由人类专家精心设计的346张图像及配对的1129个问题。我们引入了一种新颖的视觉问题结构以建立对照组，从而能够对模型的响应倾向、逻辑一致性及各类故障模式进行定量分析。在HallusionBench评估中，我们对14个不同模型进行了基准测试，结果显示最先进的GPT-4V实现了31.42%的问题对准确率。值得注意的是，所有其他评估模型的准确率均低于16%。此外，我们的分析不仅揭示了语言幻觉与视觉错觉等观测到的故障模式，还深化了对这些陷阱的理解。HallusionBench中的综合案例研究阐明了LVLM中幻觉与错觉面临的挑战。基于这些见解，我们提出了未来改进的潜在方向。该基准与代码库可访问https://github.com/tianyi-lab/HallusionBench获取。