The current research direction in generative models, such as the recently developed GPT4, aims to find relevant knowledge information for multimodal and multilingual inputs to provide answers. Under these research circumstances, the demand for multilingual evaluation of visual question answering (VQA) tasks, a representative task of multimodal systems, has increased. Accordingly, we propose a bilingual outside-knowledge VQA (BOK-VQA) dataset in this study that can be extended to multilingualism. The proposed data include 17K images, 17K question-answer pairs for both Korean and English and 280K instances of knowledge information related to question-answer content. We also present a framework that can effectively inject knowledge information into a VQA system by pretraining the knowledge information of BOK-VQA data in the form of graph embeddings. Finally, through in-depth analysis, we demonstrated the actual effect of the knowledge information contained in the constructed training data on VQA.

翻译：当前生成模型的研究方向，例如近期开发的GPT4，旨在为多模态和多语言输入寻找相关知识信息以提供答案。在此研究背景下，作为多模态系统的代表性任务，视觉问答（VQA）任务的多语言评估需求日益增长。据此，本研究提出了一个可扩展至多语言的双语外部知识VQA（BOK-VQA）数据集。所构建的数据包含17K张图像、韩语和英语各17K个问答对，以及28万个与问答内容相关的知识信息实例。我们还提出了一种框架，通过将BOK-VQA数据的知识信息以图嵌入形式进行预训练，从而有效将知识信息注入VQA系统。最后，通过深入分析，我们验证了构建的训练数据中所包含的知识信息对VQA的实际影响。