A large amount of work has been done in Multi-Agent Systems (MAS) for modeling and solving problems with multiple interacting agents. However, most LLMs are pretrained independently and not specifically optimized for coordination. Existing LLM fine-tuning frameworks rely on individual rewards, which require complex reward designs for each agent to encourage collaboration. To address these challenges, we model LLM collaboration as a cooperative Multi-Agent Reinforcement Learning (MARL) problem. We develop a multi-agent, multi-turn algorithm, Multi-Agent Group Relative Policy Optimization (MAGRPO), to solve it, building on current RL approaches for LLMs as well as MARL techniques. Our experiments on LLM writing and coding collaboration demonstrate that fine-tuning MAS with MAGRPO enables agents to generate high-quality responses efficiently through effective cooperation. Our approach opens the door to using other MARL methods for LLMs and highlights the associated challenges.

翻译：在多智能体系统（MAS）领域，已有大量工作致力于建模和解决涉及多个交互智能体的问题。然而，大多数大语言模型（LLM）在预训练阶段是独立进行的，并未针对协同合作进行专门优化。现有的大语言模型微调框架依赖于个体奖励，这需要对每个智能体设计复杂的奖励机制以促进协作。为应对这些挑战，我们将大语言模型协作建模为一个合作型多智能体强化学习（MARL）问题。基于当前面向大语言模型的强化学习方法以及多智能体强化学习技术，我们开发了一种多智能体、多轮次的算法——多智能体群组相对策略优化（MAGRPO）来解决该问题。我们在大语言模型写作与代码协作任务上的实验表明，使用MAGRPO对多智能体系统进行微调，能够使智能体通过有效合作高效生成高质量响应。我们的方法为将其他多智能体强化学习方法应用于大语言模型开辟了道路，并揭示了相关的挑战。