We introduce Wonderful Team, a multi-agent Vision Large Language Model (VLLM) framework for executing high level robotic planning in a zero-shot regime. In our context, zero-shot high-level planning means that for a novel environment, we provide a VLLM with an image of the robot's surroundings and a task description, and the VLLM outputs the sequence of actions necessary for the robot to complete the task. Unlike previous methods for high-level visual planning for robotic manipulation, our method uses VLLMs for the entire planning process, enabling a more tightly integrated loop between perception, control, and planning. As a result, Wonderful Team's performance on a real-world semantic and physical planning tasks often exceeds methods that rely on separate vision systems. For example, we see an average 40% success-rate improvement on VimaBench over prior methods such as NLaP, an average 30% improvement over Trajectory Generators on tasks from the Trajectory Generator paper including drawing and wiping a plate, and an average 70% improvement over Trajectory Generators on a new set of semantic reasoning tasks including environment re-arrangement with implicit linguistic constraints. We hope these results highlight the rapid improvements of VLLMs in the past year, and motivate the community to consider VLLMs as an option for some high-level robotic planning problems in the future.

翻译：我们提出了Wonderful Team，一种用于在零样本场景下执行高层机器人规划的多智能体视觉大语言模型（VLLM）框架。在我们的研究语境中，零样本高层规划意味着：针对全新环境，我们向VLLM提供机器人周围环境的图像及任务描述，VLLM将输出机器人完成任务所需的动作序列。与以往机器人操作的高层视觉规划方法不同，我们的方法将VLLM用于完整规划流程，实现了感知、控制与规划之间更紧密的集成闭环。因此，Wonderful Team在真实世界语义与物理规划任务中的表现通常优于依赖独立视觉系统的方法。例如，在VimaBench上我们相比NLaP等先前方法平均获得40%的成功率提升；在轨迹生成器论文涉及的绘图与擦拭餐盘等任务上，相比轨迹生成器方法平均提升30%；在包含隐式语言约束的环境重排等新型语义推理任务集上，相比轨迹生成器方法平均提升70%。我们希望这些成果能突显过去一年VLLMs的快速进步，并推动学界在未来将VLLMs视为解决某些高层机器人规划问题的可行选择。