Semantic reasoning and dynamic planning capabilities are crucial for an autonomous agent to perform complex navigation tasks in unknown environments. It requires a large amount of common-sense knowledge, that humans possess, to succeed in these tasks. We present SayNav, a new approach that leverages human knowledge from Large Language Models (LLMs) for efficient generalization to complex navigation tasks in unknown large-scale environments. SayNav uses a novel grounding mechanism, that incrementally builds a 3D scene graph of the explored environment as inputs to LLMs, for generating feasible and contextually appropriate high-level plans for navigation. The LLM-generated plan is then executed by a pre-trained low-level planner, that treats each planned step as a short-distance point-goal navigation sub-task. SayNav dynamically generates step-by-step instructions during navigation and continuously refines future steps based on newly perceived information. We evaluate SayNav on multi-object navigation (MultiON) task, that requires the agent to utilize a massive amount of human knowledge to efficiently search multiple different objects in an unknown environment. We also introduce a benchmark dataset for MultiON task employing ProcTHOR framework that provides large photo-realistic indoor environments with variety of objects. SayNav achieves state-of-the-art results and even outperforms an oracle based baseline with strong ground-truth assumptions by more than 8% in terms of success rate, highlighting its ability to generate dynamic plans for successfully locating objects in large-scale new environments. The code, benchmark dataset and demonstration videos are accessible at https://www.sri.com/ics/computer-vision/saynav.

翻译：语义推理与动态规划能力对于自主智能体在未知环境中执行复杂导航任务至关重要。这些任务需要大量人类拥有的常识性知识才能成功完成。我们提出SayNav，一种利用大语言模型（LLMs）中人类知识的新方法，可高效泛化至未知大规模环境中的复杂导航任务。SayNav采用一种新颖的具身化机制，通过逐步构建已探索环境的3D场景图作为LLMs的输入，生成可行且上下文恰当的高层导航规划。随后，由预训练的低层规划器执行LLM生成的规划，将每个规划步骤视为短距离点目标导航子任务。SayNav在导航过程中动态生成逐步指令，并基于新感知信息持续优化后续步骤。我们在多目标导航（MultiON）任务上评估SayNav——该任务要求智能体利用海量人类知识在未知环境中高效搜索多个不同物体。同时引入基于ProcTHOR框架的MultiON任务基准数据集，该框架提供包含多种物体的大型逼真室内环境。SayNav取得了最先进结果，甚至以超过8%的成功率优势超越了基于强真实假设的oracle基线，彰显其在未知大规模环境中生成动态规划以成功定位物体的能力。代码、基准数据集与演示视频访问地址：https://www.sri.com/ics/computer-vision/saynav。