We propose a novel approach to multi-robot collaboration that harnesses the power of pre-trained large language models (LLMs) for both high-level communication and low-level path planning. Robots are equipped with LLMs to discuss and collectively reason task strategies. They then generate sub-task plans and task space waypoint paths, which are used by a multi-arm motion planner to accelerate trajectory planning. We also provide feedback from the environment, such as collision checking, and prompt the LLM agents to improve their plan and waypoints in-context. For evaluation, we introduce RoCoBench, a 6-task benchmark covering a wide range of multi-robot collaboration scenarios, accompanied by a text-only dataset for agent representation and reasoning. We experimentally demonstrate the effectiveness of our approach -- it achieves high success rates across all tasks in RoCoBench and adapts to variations in task semantics. Our dialog setup offers high interpretability and flexibility -- in real world experiments, we show RoCo easily incorporates human-in-the-loop, where a user can communicate and collaborate with a robot agent to complete tasks together. See project website https://project-roco.github.io for videos and code.

翻译：摘要: 我们提出了一种新颖的多机器人协作方法，该方法利用预训练大语言模型（LLM）的强大能力，实现高层级通信与低层级路径规划。机器人配备LLM进行讨论与集体推理任务策略，随后生成子任务规划与任务空间航点路径，供多臂运动规划器加速轨迹规划。我们还提供环境反馈（如碰撞检测），并提示LLM代理在上下文中改进其规划与航点。为评估性能，我们引入RoCoBench基准测试，包含覆盖多种多机器人协作场景的6项任务，并附带用于代理表示与推理的纯文本数据集。实验证明该方法有效性——在RoCoBench所有任务中均取得高成功率，且能适应任务语义变化。我们的对话系统具有高可解释性与灵活性：在真实世界实验中，RoCo可轻松融入人类环内操作，用户能与机器人代理进行通信协作完成共同任务。视频与代码见项目网站https://project-roco.github.io。