Visual Language Models (VLMs) are powerful generative tools but often produce factually inaccurate outputs due to a lack of robust reasoning capabilities. While extensive research has been conducted on integrating external knowledge for reasoning in large language models (LLMs), such efforts remain underexplored in VLMs, where the challenge is compounded by the need to bridge multiple modalities seamlessly. This work introduces a framework for knowledge-guided reasoning in VLMs, leveraging structured knowledge graphs for multi-hop verification using image-captioning task to illustrate our framework. Our approach enables systematic reasoning across multiple steps, including visual entity recognition, knowledge graph traversal, and fact-based caption refinement. We evaluate the framework using hierarchical, triple-based and bullet-point based knowledge representations, analyzing their effectiveness in factual accuracy and logical inference. Empirical results show that our approach improves factual accuracy by approximately 31% on preliminary experiments on a curated dataset of mixtures from Google Landmarks v2, Conceptual captions and Coco captions revealing key insights into reasoning patterns and failure modes. This work demonstrates the potential of integrating external knowledge for advancing reasoning in VLMs, paving the way for more reliable and knowledgable multimodal systems.

翻译：视觉语言模型（VLMs）作为强大的生成工具，常因缺乏稳健的推理能力而产生事实性错误的输出。尽管在大语言模型（LLMs）中整合外部知识以支持推理的研究已广泛开展，此类工作在VLMs中仍显不足——其挑战因需无缝桥接多模态而更为复杂。本研究提出一种面向VLMs的知识引导推理框架，利用结构化知识图谱进行多跳验证，并以图像描述生成任务为例展示框架运作。该方法支持跨多步骤的系统性推理，包括视觉实体识别、知识图谱遍历及基于事实的描述优化。我们通过分层式、三元组式与要点式知识表示对框架进行评估，分析其在事实准确性与逻辑推理上的有效性。实验结果表明，在基于Google Landmarks v2、Conceptual Captions和COCO Captions混合构建的精选数据集上进行初步测试时，本方法将事实准确性提升了约31%，同时揭示了推理模式与失效机制的关键规律。本工作证明了整合外部知识以推进VLMs推理的潜力，为构建更可靠、更具知识深度的多模态系统奠定了基础。