The open-ended Visual Question Answering (VQA) task requires AI models to jointly reason over visual and natural language inputs using world knowledge. Recently, pre-trained Language Models (PLM) such as GPT-3 have been applied to the task and shown to be powerful world knowledge sources. However, these methods suffer from low knowledge coverage caused by PLM bias -- the tendency to generate certain tokens over other tokens regardless of prompt changes, and high dependency on the PLM quality -- only models using GPT-3 can achieve the best result. To address the aforementioned challenges, we propose RASO: a new VQA pipeline that deploys a generate-then-select strategy guided by world knowledge for the first time. Rather than following the de facto standard to train a multi-modal model that directly generates the VQA answer, RASO first adopts PLM to generate all the possible answers, and then trains a lightweight answer selection model for the correct answer. As proved in our analysis, RASO expands the knowledge coverage from in-domain training data by a large margin. We provide extensive experimentation and show the effectiveness of our pipeline by advancing the state-of-the-art by 4.1% on OK-VQA, without additional computation cost. Code and models are released at http://cogcomp.org/page/publication_view/1010

翻译：开放式视觉问答（VQA）任务要求人工智能模型利用世界知识，联合推理视觉和自然语言输入。最近，预训练语言模型（PLM），如GPT-3，已被应用于该任务，并显示出作为强大世界知识源的能力。然而，这些方法存在因PLM偏差——即无论提示如何变化，模型倾向于生成某些标记而非其他标记——导致的低知识覆盖率问题，以及对PLM质量的高度依赖性——只有使用GPT-3的模型才能达到最佳效果。为解决上述挑战，我们提出RASO：一种全新的VQA流水线，首次采用基于世界知识指导的“生成然后选择”策略。RASO并未遵循训练一个直接生成VQA答案的多模态模型的常规标准，而是首先采用PLM生成所有可能的答案，然后训练一个轻量级的答案选择模型以选出正确答案。如我们的分析所示，RASO大幅扩展了来自领域内训练数据的知识覆盖率。我们提供了广泛的实验，并通过在OK-VQA上将最新技术水平提升4.1%来展示我们流水线的有效性，且无需额外计算成本。代码和模型已在http://cogcomp.org/page/publication_view/1010上发布。