By combining natural language understanding and the generation capabilities and breadth of knowledge of large language models with image perception, recent large vision language models (LVLMs) have shown unprecedented reasoning capabilities in the real world. However, the generated text often suffers from inaccurate grounding in the visual input, resulting in errors such as hallucinating nonexistent scene elements, missing significant parts of the scene, and inferring incorrect attributes and relationships between objects. To address these issues, we introduce a novel framework, ViGoR (Visual Grounding Through Fine-Grained Reward Modeling) that utilizes fine-grained reward modeling to significantly enhance the visual grounding of LVLMs over pre-trained baselines. This improvement is efficiently achieved using much cheaper human evaluations instead of full supervisions, as well as automated methods. We show the effectiveness of our approach through numerous metrics on several benchmarks. Additionally, we construct a comprehensive and challenging dataset specifically designed to validate the visual grounding capabilities of LVLMs. Finally, we plan to release our human annotation comprising approximately 16,000 images and generated text pairs with fine-grained evaluations to contribute to related research in the community.

翻译：近年来，大型视觉语言模型通过融合自然语言理解与生成能力、大语言模型的广泛知识以及图像感知能力，在现实世界中展现出前所未有的推理能力。然而，生成的文本常出现对视觉输入定位不准确的问题，导致诸如幻觉不存在的场景元素、遗漏场景重要部分、推断对象间错误属性与关系等错误。为解决这些问题，我们提出了新型框架ViGoR（基于细粒度奖励建模的视觉定位），通过利用细粒度奖励建模，显著提升了基础预训练模型的视觉定位能力。这一改进通过更廉价的人工评估（而非完整标注）以及自动化方法高效实现。我们在多个基准测试中通过多项指标验证了方法的有效性。此外，我们构建了一个全面且具有挑战性的数据集，专门用于验证大规模视觉语言模型的视觉定位能力。最后，计划公开包含约16,000张图像及其生成文本对的人工标注数据集（附带细粒度评估），以推动该领域相关研究。