We introduce HallusionBench, a comprehensive benchmark designed for the evaluation of image-context reasoning. This benchmark presents significant challenges to advanced large visual-language models (LVLMs), such as GPT-4V(Vision) and LLaVA-1.5, by emphasizing nuanced understanding and interpretation of visual data. The benchmark comprises 346 images paired with 1129 questions, all meticulously crafted by human experts. We introduce a novel structure for these visual questions designed to establish control groups. This structure enables us to conduct a quantitative analysis of the models' response tendencies, logical consistency, and various failure modes. In our evaluation on HallusionBench, we benchmarked 13 different models, highlighting a 31.42% question-pair accuracy achieved by the state-of-the-art GPT-4V. Notably, all other evaluated models achieve accuracy below 16%. Moreover, our analysis not only highlights the observed failure modes, including language hallucination and visual illusion, but also deepens an understanding of these pitfalls. Our comprehensive case studies within HallusionBench shed light on the challenges of hallucination and illusion in LVLMs. Based on these insights, we suggest potential pathways for their future improvement. The benchmark and codebase can be accessed at https://github.com/tianyi-lab/HallusionBench.

翻译：我们提出HallusionBench，这是一个专为评估图像上下文推理能力而设计的综合性基准测试集。该基准通过强调对视觉数据的细致理解与解读，对GPT-4V(Vision)和LLaVA-1.5等先进大型视觉-语言模型（LVLMs）提出了显著挑战。该基准包含346张图片及1129道人工精心设计的问题。我们为这些视觉问题引入了一种新颖的结构，旨在建立对照组。该结构使我们能够定量分析模型的响应倾向、逻辑一致性及各类失败模式。在对HallusionBench的评估中，我们测试了13种不同模型，结果显示最先进的GPT-4V达到了31.42%的问题对准确率。值得注意的是，所有其他被评估模型的准确率均低于16%。此外，我们的分析不仅突出了包括语言幻觉和视觉错觉在内的观察到的失败模式，还深化了对这些陷阱的理解。我们在HallusionBench中的综合案例研究揭示了LVLMs中幻觉与错觉的挑战。基于这些见解，我们为其未来改进提出了潜在路径。该基准及代码库可访问https://github.com/tianyi-lab/HallusionBench。