Recent years have witnessed impressive results of pre-trained vision-language models on knowledge-intensive tasks such as visual question answering (VQA). Despite the recent advances in VQA, existing methods mainly adopt a discriminative formulation that predicts answers within a pre-defined label set, leading to easy overfitting on low-resource domains with limited labeled data (e.g., medicine) and poor generalization under domain shift to another dataset. To tackle this limitation, we propose a novel generative model enhanced by multimodal prompt retrieval (MPR) that integrates retrieved prompts and multimodal features to generate answers in free text. Our generative model enables rapid zero-shot dataset adaptation to unseen data distributions and open-set answer labels across datasets. Our experiments on medical VQA tasks show that MPR outperforms its non-retrieval counterpart by up to 30% accuracy points in a few-shot domain adaptation setting.

翻译：近年来，预训练的视觉-语言模型在知识密集型任务（如视觉问答）上取得了显著成果。尽管视觉问答技术取得进展，现有方法主要采用判别式架构，在预定义标签集内预测答案，这导致其在低资源领域（如医学）面临标注数据有限时容易过拟合，且在跨数据集领域迁移时泛化能力不足。为解决这一局限，我们提出一种基于多模态提示检索（MPR）增强的新型生成式模型，该模型通过整合检索到的提示与多模态特征，以自由文本形式生成答案。该生成式模型能够实现对新数据分布的快速零样本数据集适配，并支持跨数据集的开放集答案标签。在医学视觉问答任务上的实验表明，在小样本领域自适应场景下，MPR相比无检索框架的基线模型准确率提升高达30%。