Inspired by the superior language abilities of large language models (LLM), large vision-language models (LVLM) have been recently explored by integrating powerful LLMs for improving the performance on complex multimodal tasks. Despite the promising progress on LVLMs, we find that LVLMs suffer from the hallucination problem, i.e. they tend to generate objects that are inconsistent with the target images in the descriptions. To investigate it, this work presents the first systematic study on object hallucination of LVLMs. We conduct the evaluation experiments on several representative LVLMs, and show that they mostly suffer from severe object hallucination issue. We further discuss that the visual instructions may influence the hallucination, and find that: objects that frequently occur in the visual instructions or co-occur with the image objects, are obviously prone to be hallucinated by LVLMs. Besides, we find that existing evaluation methods might be affected by the input instructions and generation styles of LVLMs. Thus, we further design an improved evaluation method for object hallucination by proposing a polling-based query method called POPE. Experiment results demonstrate that our POPE can evaluate the object hallucination in a more stable and flexible way. Our codes and data are publicly available at https://github.com/RUCAIBox/POPE.

翻译：受大型语言模型（LLM）卓越语言能力的启发，近期通过整合强大的LLM来提升复杂多模态任务性能的大型视觉-语言模型（LVLM）得到了探索。尽管LVLM取得了令人瞩目的进展，但我们发现LVLM存在幻觉问题，即它们倾向于生成与描述中目标图像不一致的对象。为探究此问题，本文首次系统研究了LVLM的对象幻觉现象。我们针对多个代表性LVLM进行了评估实验，结果显示它们大多存在严重的对象幻觉问题。我们进一步讨论了视觉指令可能影响幻觉，并发现：在视觉指令中频繁出现或与图像对象共现的对象，明显更易被LVLM幻觉化。此外，我们发现现有评估方法可能受输入指令和LVLM生成风格的影响。因此，我们进一步设计了一种改进的对象幻觉评估方法，提出基于轮询的查询方法POPE。实验结果表明，我们的POPE能以更稳定、灵活的方式评估对象幻觉。我们的代码和数据已在https://github.com/RUCAIBox/POPE公开。