Multi-agent LLM systems enable advanced reasoning and tool use via role specialization, yet reliable reinforcement learning (RL) post-training for such systems remains difficult. In this work, we theoretically pinpoint a key reason for training instability when extending group-based RL to multi-agent LLM systems. We show that under GRPO-style optimization, a global normalization baseline may deviate from diverse agents' reward distributions, which ultimately leads to gradient-norm instability. Based on this finding, we propose Dr. MAS, a simple and stable RL training recipe for multi-agent LLM systems. Dr. MAS uses an agent-wise remedy: normalizing advantages per agent using each agent's own reward statistics, which calibrates gradient scales and dramatically stabilizes training, both theoretically and empirically. Beyond the algorithm, Dr. MAS provides an end-to-end RL training framework for multi-agent LLM systems, supporting scalable orchestration, flexible per-agent LLM serving and optimization configs, and shared resource scheduling of LLM actor backends. We evaluate Dr. MAS on multi-agent math reasoning and multi-turn search benchmarks using Qwen2.5 and Qwen3 series models. Dr. MAS achieves clear gains over vanilla GRPO (e.g., +5.6\% avg@16 and +4.6\% pass@16 on math, and +15.2\% avg@16 and +13.1\% pass@16 on search) while largely eliminating gradient spikes. Moreover, it remains highly effective under heterogeneous agent-model assignments while improving efficiency.

翻译：多智能体大语言模型系统通过角色分工实现了高级推理与工具调用，然而为此类系统进行可靠的强化学习后训练仍具挑战性。本文从理论上指出了将基于群体的强化学习扩展至多智能体大语言模型系统时，训练不稳定的一个关键原因。我们证明，在GRPO风格的优化下，全局归一化基线可能偏离不同智能体的奖励分布，最终导致梯度范数不稳定。基于这一发现，我们提出了Dr. MAS，一种面向多智能体大语言模型系统的简单且稳定的强化学习训练方案。Dr. MAS采用了一种按智能体调整的补救措施：利用每个智能体自身的奖励统计量对其优势函数进行归一化，从而校准梯度尺度，并在理论和实验上显著稳定了训练。除了算法本身，Dr. MAS还为多智能体大语言模型系统提供了一个端到端的强化学习训练框架，支持可扩展的编排、灵活的按智能体大语言模型服务与优化配置，以及大语言模型执行器后端的共享资源调度。我们在多智能体数学推理和多轮搜索基准测试上，使用Qwen2.5和Qwen3系列模型对Dr. MAS进行了评估。Dr. MAS相比原始GRPO取得了显著提升（例如，在数学任务上平均提升5.6%（avg@16）和4.6%（pass@16），在搜索任务上平均提升15.2%（avg@16）和13.1%（pass@16）），同时基本消除了梯度尖峰。此外，即使在异构的智能体-模型分配下，它仍能保持高效且有效。