Scaling reinforcement learning to tens of thousands of parallel environments requires overcoming the limited exploration capacity of a single policy. Ensemble-based policy gradient methods, which employ multiple policies to collect diverse samples, have recently been proposed to promote exploration. However, merely broadening the exploration space does not always enhance learning capability, since excessive exploration can reduce exploration quality or compromise training stability. In this work, we theoretically analyze the impact of inter-policy diversity on learning efficiency in policy ensembles, and propose Coupled Policy Optimization which regulates diversity through KL constraints between policies. The proposed method enables effective exploration and outperforms strong baselines such as SAPG, PBT, and PPO across multiple tasks, including challenging dexterous manipulation, in terms of both sample efficiency and final performance. Furthermore, analysis of policy diversity and effective sample size during training reveals that follower policies naturally distribute around the leader, demonstrating the emergence of structured and efficient exploratory behavior. Our results indicate that diverse exploration under appropriate regulation is key to achieving stable and sample-efficient learning in ensemble policy gradient methods. Project page at https://naoki04.github.io/paper-cpo/ .

翻译：将强化学习扩展到数万个并行环境需要克服单一策略有限的探索能力。最近提出的基于集成的策略梯度方法通过采用多个策略收集多样化样本来促进探索。然而，单纯扩大探索空间并不总能提升学习能力，因为过度探索可能降低探索质量或损害训练稳定性。本文从理论上分析了策略间多样性对策略集成学习效率的影响，并提出通过策略间KL约束调控多样性的耦合策略优化方法。所提出的方法实现了有效探索，在包括挑战性灵巧操作在内的多个任务中，在样本效率和最终性能方面均优于SAPG、PBT和PPO等强基线方法。此外，对训练过程中策略多样性和有效样本量的分析表明，跟随策略会自然地围绕主导策略分布，这体现了结构化高效探索行为的涌现。我们的研究结果表明，在适当调控下的多样化探索是实现集成策略梯度方法稳定且样本高效学习的关键。项目页面：https://naoki04.github.io/paper-cpo/。