Grounding the reasoning ability of large language models (LLMs) for embodied tasks is challenging due to the complexity of the physical world. Especially, LLM planning for multi-agent collaboration requires communication of agents or credit assignment as the feedback to re-adjust the proposed plans and achieve effective coordination. However, existing methods that overly rely on physical verification or self-reflection suffer from excessive and inefficient querying of LLMs. In this paper, we propose a novel framework for multi-agent collaboration that introduces Reinforced Advantage feedback (ReAd) for efficient self-refinement of plans. Specifically, we perform critic regression to learn a sequential advantage function from LLM-planned data, and then treat the LLM planner as an optimizer to generate actions that maximize the advantage function. It endows the LLM with the foresight to discern whether the action contributes to accomplishing the final task. We provide theoretical analysis by extending advantage-weighted regression in reinforcement learning to multi-agent systems. Experiments on Overcooked-AI and a difficult variant of RoCoBench show that ReAd surpasses baselines in success rate, and also significantly decreases the interaction steps of agents and query rounds of LLMs, demonstrating its high efficiency for grounding LLMs. More results are given at \url{https://read-llm.github.io/}.

翻译：由于物理世界的复杂性，将大型语言模型（LLM）的推理能力应用于具身任务极具挑战。特别是，面向多智能体协作的LLM规划需要智能体间的通信或信用分配作为反馈，以重新调整所提出的计划并实现有效协调。然而，现有方法过度依赖物理验证或自我反思，导致对LLM的查询次数过多且效率低下。本文提出一种用于多智能体协作的新型框架，该框架引入了强化优势反馈（ReAd）以实现计划的高效自我优化。具体而言，我们通过评论家回归从LLM规划的数据中学习一个序列优势函数，然后将LLM规划器视为优化器，以生成最大化该优势函数的动作。这赋予了LLM一种预见能力，使其能够判断动作是否有助于完成最终任务。我们通过将强化学习中的优势加权回归扩展到多智能体系统，提供了理论分析。在Overcooked-AI和RoCoBench的一个困难变体上的实验表明，ReAd在成功率上超越了基线方法，并且显著减少了智能体的交互步数和LLM的查询轮次，证明了其在LLM落地应用方面的高效性。更多结果请访问 \url{https://read-llm.github.io/}。