Recent efforts to enable visual navigation using large language models have mainly focused on developing complex prompt systems. These systems incorporate instructions, observations, and history into massive text prompts, which are then combined with pre-trained large language models to facilitate visual navigation. In contrast, our approach aims to fine-tune large language models for visual navigation without extensive prompt engineering. Our design involves a simple text prompt, current observations, and a history collector model that gathers information from previous observations as input. For output, our design provides a probability distribution of possible actions that the agent can take during navigation. We train our model using human demonstrations and collision signals from the Habitat-Matterport 3D Dataset (HM3D). Experimental results demonstrate that our method outperforms state-of-the-art behavior cloning methods and effectively reduces collision rates.

翻译：近期，利用大语言模型实现视觉导航的研究主要集中于开发复杂的提示系统。这些系统将指令、观测结果和历史信息整合到大规模文本提示中，随后与预训练的大语言模型结合，以促进视觉导航。相比之下，我们的方法旨在直接微调大语言模型用于视觉导航，无需复杂的提示工程。我们的设计包括一个简单的文本提示、当前观测结果，以及一个从先前观测中收集信息的历史收集器模型作为输入。输出方面，我们的设计提供了智能体在导航过程中可采取动作的概率分布。我们利用人类演示数据和来自Habitat-Matterport 3D数据集（HM3D）的碰撞信号训练模型。实验结果表明，我们的方法优于当前最先进的行为克隆方法，并能有效降低碰撞率。