Large language models have demonstrated an emergent capability in answering knowledge intensive questions. With recent progress on web-scale visual and language pre-training, do these models also understand how to answer visual information seeking questions? To answer this question, we present InfoSeek, a Visual Question Answering dataset that focuses on asking information-seeking questions, where the information can not be answered by common sense knowledge. We perform a multi-stage human annotation to collect a natural distribution of high-quality visual information seeking question-answer pairs. We also construct a large-scale, automatically collected dataset by combining existing visual entity recognition datasets and Wikidata, which provides over one million examples for model fine-tuning and validation. Based on InfoSeek, we analyzed various pre-trained Visual QA systems to gain insights into the characteristics of different pre-trained models. Our analysis shows that it is challenging for the state-of-the-art multi-modal pre-trained models to answer visual information seeking questions, but this capability is improved through fine-tuning on the automated InfoSeek dataset. We hope our analysis paves the way to understand and develop the next generation of multi-modal pre-training.

翻译：大型语言模型已展现出回答知识密集型问题的涌现能力。随着大规模视觉与语言预训练技术的进展，这些模型是否也能理解如何回答视觉信息查询问题？为解答此问题，我们提出了InfoSeek——一个聚焦于信息查询类问题的视觉问答数据集，其中所需信息无法通过常识回答。通过多阶段人工标注，我们收集了具有自然分布的高质量视觉信息查询问答对。同时，我们结合现有视觉实体识别数据集与维基百科，构建了大规模自动采集数据集，为模型微调与验证提供逾百万样本。基于InfoSeek，我们分析了多种预训练视觉问答系统，深入探究不同预训练模型的特性。分析表明，当前最先进的多模态预训练模型在回答视觉信息查询问题时仍面临挑战，但通过基于自动化InfoSeek数据集的微调可提升该能力。我们期望此分析能为理解与开发下一代多模态预训练模型奠定基础。