Knowledge-based visual question answering is a very challenging and widely concerned task. Previous methods adopts the implicit knowledge in large language models (LLM) to achieve excellent results, but we argue that existing methods may suffer from biasing understanding of the image and insufficient knowledge to solve the problem. In this paper, we propose PROOFREAD -PROmpting vision language model with knOwledge From laRgE lAnguage moDel, a novel, lightweight and efficient kowledge-based VQA framework, which make the vision language model and the large language model cooperate to give full play to their respective strengths and bootstrap each other. In detail, our proposed method uses LLM to obtain knowledge explicitly, uses the vision language model which can see the image to get the knowledge answer, and introduces knowledge perceiver to filter out knowledge that is harmful for getting the correct final answer. Experimental results on two datasets prove the effectiveness of our approach. Our method outperforms all state-of-the-art methods on the A-OKVQA dataset in two settings and also achieves relatively good performance on the OKVQA dataset.

翻译：基于知识的视觉问答是一项极具挑战性且备受关注的任务。以往方法借助大语言模型中的隐式知识取得了优异成果，但我们认为现有方法可能面临对图像理解的偏差以及解决问题时知识不足的问题。本文提出PROOFREAD——一种利用大语言模型知识提示视觉语言模型的新颖、轻量且高效的基于知识的VQA框架，该框架使视觉语言模型与大语言模型协同工作，充分发挥各自优势并相互促进。具体而言，我们提出的方法显式地通过LLM获取知识，利用具备图像感知能力的视觉语言模型获取知识答案，并引入知识感知器来过滤对获得正确最终答案有害的知识。两个数据集上的实验结果证明了我们方法的有效性。在A-OKVQA数据集上，我们的方法在两种设置下均超越了所有现有最先进方法，并在OKVQA数据集上也取得了相对较好的性能。