Multi-Agent Reinforcement Learning (MARL) provides a powerful framework for learning coordination in multi-agent systems. However, applying MARL to robotics remains challenging due to their high-dimensional continuous joint action spaces, complex reward design, and non-stationarity from concurrently learning agents. On the other hand, humans often learn complex coordination with the help of coaches, who guide learning through carefully designed curricula and detailed feedback. Building on the reasoning capabilities of foundation models, we argue that these models can similarly coach robots to learn coordination. Motivated by this, we propose CRAFT: Coaching Reinforcement learning Autonomously using Foundation models for learning coordination Tasks, a framework that leverages foundation models to act as a "coach" for multi-robot coordination. CRAFT automatically decomposes long-horizon coordination tasks into sequences of subtasks using the planning capability of Large Language Models (LLMs). Then, CRAFT trains each subtask using LLM-generated reward functions, and refines them through a Vision Language Model (VLM)-guided reward-refinement loop. We evaluate CRAFT on multi-quadruped navigation and bimanual manipulation tasks, and demonstrate its capability to learn complex coordination behaviors. In addition, in a multi-quadruped navigation setting, we show that our learned policies transfer to the real world. Project website is https://iconlab.negarmehr.com/CRAFT/

翻译：摘要：多智能体强化学习（MARL）为多智能体系统中的协调学习提供了强大的框架。然而，由于机器人领域存在高维连续联合动作空间、复杂的奖励设计以及并发学习智能体导致的非平稳性，将MARL应用于机器人仍具挑战性。另一方面，人类往往借助教练的指导来学习复杂的协调技能——教练通过精心设计的课程和详细反馈引导学习过程。基于基础模型的推理能力，我们认为这些模型可以类似地引导机器人学习协调技能。受此启发，我们提出CRAFT（基于基础模型自主进行协调任务强化学习辅导）框架，利用基础模型充当多机器人协调的“教练”。CRAFT首先利用大语言模型（LLMs）的规划能力，将长时域协调任务自动分解为子任务序列。随后，通过LLM生成的奖励函数训练每个子任务，并借助视觉语言模型（VLM）引导的奖励精化循环对奖励进行优化。我们在多四足机器人导航和双臂操作任务上评估了CRAFT，证明了其学习复杂协调行为的能力。此外，在多四足机器人导航场景中，我们展示了学得的策略可迁移至真实世界。项目网站：https://iconlab.negarmehr.com/CRAFT/