Humans have a natural ability to perform semantic associations with the surrounding objects in the environment. This allows them to create a mental map of the environment which helps them to navigate on-demand when given a linguistic instruction. A natural goal in Vision Language Navigation (VLN) research is to impart autonomous agents with similar capabilities. Recently introduced VL Maps \cite{huang23vlmaps} take a step towards this goal by creating a semantic spatial map representation of the environment without any labelled data. However, their representations are limited for practical applicability as they do not distinguish between different instances of the same object. In this work, we address this limitation by integrating instance-level information into spatial map representation using a community detection algorithm and by utilizing word ontology learned by large language models (LLMs) to perform open-set semantic associations in the mapping representation. The resulting map representation improves the navigation performance by two-fold (233\%) on realistic language commands with instance-specific descriptions compared to VL Maps. We validate the practicality and effectiveness of our approach through extensive qualitative and quantitative experiments.

翻译：人类天生具备对环境中的周围对象进行语义关联的能力。这使他们能够构建环境的心理地图，从而在接收语言指令时按需导航。视觉语言导航（VLN）研究的一个自然目标是赋予自主代理类似的能力。最近提出的VL Maps \cite{huang23vlmaps}通过在没有标注数据的情况下创建环境的语义空间地图表示，朝着这一目标迈出了一步。然而，由于它们不区分同一对象的不同实例，其表示在实际应用中受到限制。在这项工作中，我们通过使用社区检测算法将实例级信息整合到空间地图表示中，并利用大型语言模型（LLM）学习的词本体在映射表示中执行开放集语义关联，来解决这一限制。与VL Maps相比，所得到的地图表示在包含实例特定描述的逼真语言命令上，导航性能提升了两倍（233%）。我们通过大量的定性和定量实验验证了我们方法的实用性和有效性。