Advancements in large language models (LLMs) have demonstrated their potential in facilitating high-level reasoning, logical reasoning and robotics planning. Recently, LLMs have also been able to generate reward functions for low-level robot actions, effectively bridging the interface between high-level planning and low-level robot control. However, the challenge remains that even with syntactically correct plans, robots can still fail to achieve their intended goals. This failure can be attributed to imperfect plans proposed by LLMs or to unforeseeable environmental circumstances that hinder the execution of planned subtasks due to erroneous assumptions about the state of objects. One way to prevent these challenges is to rely on human-provided step-by-step instructions, limiting the autonomy of robotic systems. Vision Language Models (VLMs) have shown remarkable success in tasks such as visual question answering and image captioning. Leveraging the capabilities of VLMs, we present a novel framework called Robotic Replanning with Perception and Language Models (RePLan) that enables real-time replanning capabilities for long-horizon tasks. This framework utilizes the physical grounding provided by a VLM's understanding of the world's state to adapt robot actions when the initial plan fails to achieve the desired goal. We test our approach within four environments containing seven long-horizion tasks. We find that RePLan enables a robot to successfully adapt to unforeseen obstacles while accomplishing open-ended, long-horizon goals, where baseline models cannot. Find more information at https://replan-lm.github.io/replan.github.io/

翻译：摘要：大型语言模型（LLMs）的进步已证明其在促进高级推理、逻辑推理和机器人规划方面的潜力。近年来，LLMs还能生成用于低级机器人动作的奖励函数，有效衔接了高级规划与低级机器人控制之间的接口。然而，即便规划在语法上正确，机器人仍可能无法达成预期目标，这一挑战依然存在。这种失败可归因于LLMs提出的不完美规划，或由于对物体状态的错误假设导致不可预见的环境状况阻碍了计划子任务的执行。应对这些挑战的一种方式依赖于人类提供的逐步指令，但这限制了机器人系统的自主性。视觉语言模型（VLMs）在视觉问答和图像描述等任务中展现出显著成功。利用VLMs的能力，我们提出了一种名为“基于感知与语言模型的机器人重规划”（RePLan）的新框架，该框架为长周期任务实现了实时重规划能力。此框架利用VLM对世界状态理解的物理基础，在初始规划未能实现预期目标时调整机器人动作。我们在包含七个长周期任务的四个环境中测试了该方法。研究发现，RePLan使机器人能够成功适应不可预见的障碍，同时完成开放式的长周期目标，而基线模型无法做到这一点。更多信息请访问https://replan-lm.github.io/replan.github.io/