Following language instructions to navigate in unseen environments is a challenging task for autonomous embodied agents. With strong representation capabilities, pretrained vision-and-language models are widely used in VLN. However, most of them are trained on web-crawled general-purpose datasets, which incurs a considerable domain gap when used for VLN tasks. To address the problem, we propose a novel and model-agnostic domain-aware prompt learning (DAP) framework. For equipping the pretrained models with specific object-level and scene-level cross-modal alignment in VLN tasks, DAP applies a low-cost prompt tuning paradigm to learn soft visual prompts for extracting in-domain image semantics. Specifically, we first generate a set of in-domain image-text pairs with the help of the CLIP model. Then we introduce soft visual prompts in the input space of the visual encoder in a pretrained model. DAP injects in-domain visual knowledge into the visual encoder of the pretrained model in an efficient way. Experimental results on both R2R and REVERIE show the superiority of DAP compared to existing state-of-the-art methods.

翻译：摘要：根据语言指令在未知环境中导航对于自主具身智能体而言是一项具有挑战性的任务。凭借强大的表示能力，预训练的视觉与语言模型被广泛应用于视觉语言导航（VLN）任务中。然而，这些模型大多基于网络爬取的通用数据集进行训练，当应用于VLN任务时会产生显著领域差距。为解决这一问题，我们提出了一种新颖且与模型无关的领域感知提示学习（DAP）框架。为了使预训练模型在VLN任务中具备特定的物体级与场景级跨模态对齐能力，DAP采用低成本的提示调优范式，通过学习软视觉提示来提取域内图像语义。具体而言，我们首先借助CLIP模型生成一组域内图像-文本对，随后在预训练模型的视觉编码器输入空间中引入软视觉提示。DAP通过高效方式将域内视觉知识注入预训练模型的视觉编码器。在R2R和REVERIE数据集上的实验结果表明，与现有最先进方法相比，DAP具有显著优越性。