Knowledge-based visual question answering (VQA) involves questions that require world knowledge beyond the image to yield the correct answer. Large language models (LMs) like GPT-3 are particularly helpful for this task because of their strong knowledge retrieval and reasoning capabilities. To enable LM to understand images, prior work uses a captioning model to convert images into text. However, when summarizing an image in a single caption sentence, which visual entities to describe are often underspecified. Generic image captions often miss visual details essential for the LM to answer visual questions correctly. To address this challenge, we propose PromptCap (Prompt-guided image Captioning), a captioning model designed to serve as a better connector between images and black-box LMs. Different from generic captions, PromptCap takes a natural-language prompt to control the visual entities to describe in the generated caption. The prompt contains a question that the caption should aid in answering. To avoid extra annotation, PromptCap is trained by examples synthesized with GPT-3 and existing datasets. We demonstrate PromptCap's effectiveness on an existing pipeline in which GPT-3 is prompted with image captions to carry out VQA. PromptCap outperforms generic captions by a large margin and achieves state-of-the-art accuracy on knowledge-based VQA tasks (60.4% on OK-VQA and 59.6% on A-OKVQA). Zero-shot results on WebQA show that PromptCap generalizes well to unseen domains.

翻译：基于知识的视觉问答（VQA）涉及需要超越图像本身的世界知识才能得出正确答案的问题。大型语言模型（LM），如GPT-3，因其强大的知识检索和推理能力，在此任务中尤为有用。为使LM能够理解图像，先前的工作使用描述模型将图像转换为文本。然而，当用单一描述句概括图像时，应描述哪些视觉实体往往定义不清。通用图像描述常常遗漏LM正确回答视觉问题所必需的视觉细节。为解决这一挑战，我们提出PromptCap（提示引导的图像描述），这是一个旨在更好地连接图像与黑盒LM的描述模型。与通用描述不同，PromptCap利用自然语言提示来控制生成描述中需要描述的视觉实体。该提示包含一个有助于回答的描述问题。为避免额外标注，PromptCap通过GPT-3和现有数据集合成的示例进行训练。我们在现有流程中证明了PromptCap的有效性，其中GPT-3通过图像描述被提示以执行VQA。PromptCap在性能上大幅优于通用描述，并在基于知识的VQA任务上达到了最先进的准确率（OK-VQA上为60.4%，A-OKVQA上为59.6%）。在WebQA上的零样本结果显示，PromptCap能很好地泛化到未见领域。