Semantic reasoning and dynamic planning capabilities are crucial for an autonomous agent to perform complex navigation tasks in unknown environments. It requires a large amount of common-sense knowledge, that humans possess, to succeed in these tasks. We present SayNav, a new approach that leverages human knowledge from Large Language Models (LLMs) for efficient generalization to complex navigation tasks in unknown large-scale environments. SayNav uses a novel grounding mechanism, that incrementally builds a 3D scene graph of the explored environment as inputs to LLMs, for generating feasible and contextually appropriate high-level plans for navigation. The LLM-generated plan is then executed by a pre-trained low-level planner, that treats each planned step as a short-distance point-goal navigation sub-task. SayNav dynamically generates step-by-step instructions during navigation and continuously refines future steps based on newly perceived information. We evaluate SayNav on a new multi-object navigation task, that requires the agent to utilize a massive amount of human knowledge to efficiently search multiple different objects in an unknown environment. SayNav outperforms an oracle based Point-nav baseline, achieving a success rate of 95.35% (vs 56.06% for the baseline), under the ideal settings on this task, highlighting its ability to generate dynamic plans for successfully locating objects in large-scale new environments. In addition, SayNav also enables efficient generalization from simulation to real environments.

翻译：语义推理与动态规划能力对于自主智能体在未知环境中执行复杂导航任务至关重要。这类任务需要大量人类具备的常识知识才能成功完成。我们提出SayNav——一种利用大语言模型（LLMs）中人类知识的新方法，可高效泛化至未知大规模环境中的复杂导航任务。SayNav采用新型具身化机制，通过逐步构建已探索环境的三维场景图作为LLMs输入，生成可行且上下文适配的高层级导航规划。LLM生成的规划随后由预训练的底层规划器执行，该规划器将每个规划步骤视为短距离点目标导航子任务。SayNav在导航过程中动态生成逐步指令，并基于新感知信息持续优化后续步骤。我们在新型多目标导航任务上评估SayNav，该任务要求智能体利用海量人类知识在未知环境中高效搜索多个不同物体。在该任务理想设定下，SayNav超越基于点导航的oracle基线，达到95.35%的成功率（基线为56.06%），突显其在大型新环境中生成动态规划以成功定位物体的能力。此外，SayNav还能实现从仿真环境到真实环境的高效泛化。