Within the multimodal field, large vision-language models (LVLMs) have made significant progress due to their strong perception and reasoning capabilities in the visual and language systems. However, LVLMs are still plagued by the two critical issues of object hallucination and factual accuracy, which limit the practicality of LVLMs in different scenarios. Furthermore, previous evaluation methods focus more on the comprehension and reasoning of language content but lack a comprehensive evaluation of multimodal interactions, thereby resulting in potential limitations. To this end, we propose a novel KNVQA-Eval, which is devoted to knowledge-based VQA task evaluation to reflect the factuality of multimodal LVLMs. To ensure the robustness and scalability of the evaluation, we develop a new KNVQA dataset by incorporating human judgment and perception, aiming to evaluate the accuracy of standard answers relative to AI-generated answers in knowledge-based VQA. This work not only comprehensively evaluates the contextual information of LVLMs using reliable human annotations, but also further analyzes the fine-grained capabilities of current methods to reveal potential avenues for subsequent optimization of LVLMs-based estimators. Our proposed VQA-Eval and corresponding dataset KNVQA will facilitate the development of automatic evaluation tools with the advantages of low cost, privacy protection, and reproducibility. Our code will be released upon publication.

翻译：在多模态领域，大型视觉语言模型（LVLMs）因其在视觉和语言系统中的强大感知与推理能力而取得了显著进展。然而，LVLMs仍受制于物体幻觉与事实准确性这两个关键问题，这限制了它们在不同场景中的实用性。此外，以往的评估方法更侧重于语言内容的理解与推理，缺乏对多模态交互的全面评估，因而存在潜在局限性。为此，我们提出了一种新的KNVQA-Eval，致力于基于知识的VQA任务评估，以反映多模态LVLMs的事实性。为确保评估的鲁棒性与可扩展性，我们通过融入人类判断与感知，开发了新的KNVQA数据集，旨在评估基于知识的VQA中标准答案相对于AI生成答案的准确性。这项工作不仅利用可靠的人类标注全面评估了LVLMs的上下文信息，还进一步分析了当前方法的细粒度能力，为后续优化基于LVLMs的评估工具揭示了潜在途径。我们提出的VQA-Eval及相应的KNVQA数据集，将推动具有低成本、隐私保护与可复现性优势的自动评估工具的发展。代码将在论文发表后公开。