Object hallucination poses a significant challenge in vision-language (VL) models, often leading to the generation of nonsensical or unfaithful responses with non-existent objects. However, the absence of a general measurement for evaluating object hallucination in VL models has hindered our understanding and ability to mitigate this issue. In this work, we present NOPE (Negative Object Presence Evaluation), a novel benchmark designed to assess object hallucination in VL models through visual question answering (VQA). We propose a cost-effective and scalable approach utilizing large language models to generate 29.5k synthetic negative pronoun (NegP) data of high quality for NOPE. We extensively investigate the performance of 10 state-of-the-art VL models in discerning the non-existence of objects in visual questions, where the ground truth answers are denoted as NegP (e.g., "none"). Additionally, we evaluate their standard performance on visual questions on 9 other VQA datasets. Through our experiments, we demonstrate that no VL model is immune to the vulnerability of object hallucination, as all models achieve accuracy below 10\% on NegP. Furthermore, we uncover that lexically diverse visual questions, question types with large scopes, and scene-relevant objects capitalize the risk of object hallucination in VL models.

翻译：物体幻觉是视觉语言（VL）模型面临的一个重大挑战，常导致模型生成包含不存在物体的无意义或不忠实回应。然而，由于缺乏衡量VL模型物体幻觉的通用评估方法，我们对此问题的理解及缓解能力受到阻碍。本文提出NOPE（负性物体存在评估），一种通过视觉问答（VQA）评估VL模型物体幻觉的新型基准。我们采用一种高性价比且可扩展的方法，利用大语言模型生成29.5k高质量合成负代词（NegP）数据用于NOPE。我们深入研究了10个最先进VL模型在视觉问题中判断物体不存在性的表现（其真实答案标注为NegP，例如“无”），并评估了它们在9个其他VQA数据集上的标准视觉问题表现。实验表明，所有VL模型均无法避免物体幻觉的脆弱性——所有模型在NegP上的准确率均低于10%。此外，我们发现词汇多样性视觉问题、大范围问题类型以及与场景相关的物体会显著增加VL模型发生物体幻觉的风险。