We analyze knowledge-based visual question answering, for which given a question, the models need to ground it into the visual modality and retrieve the relevant knowledge from a given large knowledge base (KB) to be able to answer. Our analysis has two folds, one based on designing neural architectures and training them from scratch, and another based on large pre-trained language models (LLMs). Our research questions are: 1) Can we effectively augment models by explicit supervised retrieval of the relevant KB information to solve the KB-VQA problem? 2) How do task-specific and LLM-based models perform in the integration of visual and external knowledge, and multi-hop reasoning over both sources of information? 3) Is the implicit knowledge of LLMs sufficient for KB-VQA and to what extent it can replace the explicit KB? Our results demonstrate the positive impact of empowering task-specific and LLM models with supervised external and visual knowledge retrieval models. Our findings show that though LLMs are stronger in 1-hop reasoning, they suffer in 2-hop reasoning in comparison with our fine-tuned NN model even if the relevant information from both modalities is available to the model. Moreover, we observed that LLM models outperform the NN model for KB-related questions which confirms the effectiveness of implicit knowledge in LLMs however, they do not alleviate the need for external KB.

翻译：我们分析了基于知识的视觉问答任务，在此任务中，模型需要将问题与视觉模态对齐，并从给定的大型知识库（KB）中检索相关知识以给出答案。本研究包含两个方向：一是设计神经网络架构并进行端到端训练，二是基于大型预训练语言模型（LLM）。研究问题包括：1）能否通过显式监督式检索知识库相关信息，有效增强模型以解决KB-VQA问题？2）在融合视觉信息与外部知识，并对两类信息进行多跳推理时，任务专用模型与LLM模型的表现如何？3）LLM的内隐知识是否足以解决KB-VQA任务，以及在何种程度上可替代显式知识库？实验结果表明，通过监督式外部知识与视觉知识检索模型增强任务专用模型与LLM模型具有积极效果。研究发现：即使模型能同时获取两类模态的相关信息，LLM在1跳推理上表现更强，但在2跳推理上弱于我们微调的神经网络模型。此外，我们观察到LLM在涉及知识库的问题上表现优于神经网络模型，证实了LLM内隐知识的有效性，但这类模型仍无法消除对外部知识库的依赖。