Grounding the reasoning ability of large language models (LLMs) for embodied tasks is challenging due to the complexity of the physical world. Especially, LLM planning for multi-agent collaboration requires communication of agents or credit assignment as the feedback to re-adjust the proposed plans and achieve effective coordination. However, existing methods that overly rely on physical verification or self-reflection suffer from excessive and inefficient querying of LLMs. In this paper, we propose a novel framework for multi-agent collaboration that introduces Reinforced Advantage feedback (ReAd) for efficient self-refinement of plans. Specifically, we perform critic regression to learn a sequential advantage function from LLM-planned data, and then treat the LLM planner as an optimizer to generate actions that maximize the advantage function. It endows the LLM with the foresight to discern whether the action contributes to accomplishing the final task. We provide theoretical analysis by extending advantage-weighted regression in reinforcement learning to multi-agent systems. Experiments on Overcooked-AI and a difficult variant of RoCoBench show that ReAd surpasses baselines in success rate, and also significantly decreases the interaction steps of agents and query rounds of LLMs, demonstrating its high efficiency for grounding LLMs. More results are given at https://read-llm.github.io/.

翻译：将大型语言模型（LLM）的推理能力应用于具身任务时，由于物理世界的复杂性，其实体化过程面临挑战。特别是在多智能体协作的LLM规划中，需要智能体间的通信或信用分配作为反馈，以重新调整所提出的计划并实现有效协调。然而，现有方法过度依赖物理验证或自我反思，导致对LLM的查询次数过多且效率低下。本文提出一种新颖的多智能体协作框架，引入强化优势反馈（ReAd）以实现计划的高效自我优化。具体而言，我们通过评论家回归从LLM规划数据中学习序列优势函数，然后将LLM规划器视为优化器，以生成最大化该优势函数的动作。这赋予LLM前瞻能力，使其能够判别动作是否有助于最终任务的完成。我们通过将强化学习中的优势加权回归扩展至多智能体系统提供了理论分析。在Overcooked-AI和RoCoBench的困难变体上的实验表明，ReAd在成功率上超越基线方法，并显著减少了智能体的交互步数和LLM的查询轮次，证明了其在LLM实体化方面的高效性。更多结果请访问 https://read-llm.github.io/。