Equipping embodied agents with commonsense is important for robots to successfully complete complex human instructions in general environments. Recent large language models (LLM) can embed rich semantic knowledge for agents in plan generation of complex tasks, while they lack the information about the realistic world and usually yield infeasible action sequences. In this paper, we propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning with physical scene constraint, where the agent generates executable plans according to the existed objects in the scene by aligning LLMs with the visual perception models. Specifically, we first construct a multimodal dataset containing triplets of indoor scenes, instructions and action plans, where we provide the designed prompts and the list of existing objects in the scene for GPT-3.5 to generate a large number of instructions and corresponding planned actions. The generated data is leveraged for grounded plan tuning of pre-trained LLMs. During inference, we discover the objects in the scene by extending open-vocabulary object detectors to multi-view RGB images collected in different achievable locations. Experimental results show that the generated plan from our TaPA framework can achieve higher success rate than LLaVA and GPT-3.5 by a sizable margin, which indicates the practicality of embodied task planning in general and complex environments.

翻译：赋予具身代理常识对于机器人在通用环境中成功完成复杂的人类指令至关重要。近期的大语言模型（LLM）能够为代理在复杂任务规划中嵌入丰富的语义知识，但由于缺乏现实世界信息，常生成不可行的动作序列。本文提出了一种具身任务中的任务规划代理（TaPA），用于在物理场景约束下进行有依据的规划——该代理通过将大语言模型与视觉感知模型对齐，根据场景中存在的物体生成可执行规划。具体而言，我们首先构建了一个包含室内场景、指令和动作规划三元组的多模态数据集，利用设计的提示词和场景中现有物体列表驱动GPT-3.5生成大量指令及其对应的规划动作。生成数据被用于预训练语言模型的有依据规划微调。在推理阶段，我们通过将开放词汇目标检测器扩展到不同可达位置采集的多视角RGB图像，实现场景中物体的发现。实验结果表明，我们的TaPA框架生成的规划在成功率上显著高于LLaVA和GPT-3.5，这体现了具身任务规划在通用和复杂环境中的实用性。