Large Vision-Language Models (LVLMs) suffer from hallucination issues, wherein the models generate plausible-sounding but factually incorrect outputs, undermining their reliability. A comprehensive quantitative evaluation is necessary to identify and understand the extent of hallucinations in these models. However, existing benchmarks are often limited in scope, focusing mainly on object hallucinations. Furthermore, current evaluation methods struggle to effectively address the subtle semantic distinctions between model outputs and reference data, as well as the balance between hallucination and informativeness. To address these issues, we introduce a multi-dimensional benchmark covering objects, attributes, and relations, with challenging images selected based on associative biases. Moreover, we propose an large language model (LLM)-based two-stage evaluation framework that generalizes the popular CHAIR metric and incorporates both faithfulness and coverage into the evaluation. Experiments on 10 established LVLMs demonstrate that our evaluation metric is more comprehensive and better correlated with humans than existing work when evaluating on our challenging human annotated benchmark dataset. Our work also highlights the critical balance between faithfulness and coverage of model outputs, and encourages future works to address hallucinations in LVLMs while keeping their outputs informative.

翻译：大型视觉-语言模型（LVLMs）存在幻觉问题，即模型会生成看似合理但事实错误的输出，从而削弱其可靠性。为识别并理解这些模型中幻觉的程度，亟需进行全面的量化评估。然而，现有基准测试范围往往有限，主要聚焦于对象幻觉。此外，当前评估方法难以有效处理模型输出与参考数据之间的细微语义差异，以及幻觉与信息量之间的权衡。为解决这些问题，我们引入了一个涵盖对象、属性和关系的多维度基准测试，并基于关联性偏差选取具有挑战性的图像。同时，我们提出了一种基于大语言模型（LLM）的两阶段评估框架，该框架泛化了流行的CHAIR指标，并将忠实性与覆盖性纳入评估。对10个已建立的LVLMs的实验表明，在评估我们具有挑战性的人工标注基准数据集时，我们的评估指标比现有工作更全面，且与人类判断的相关性更高。我们的工作还突显了模型输出的忠实性与覆盖性之间的关键平衡，并鼓励未来研究在保持模型输出信息丰富性的同时，解决LVLMs中的幻觉问题。