Curriculum reinforcement learning (CRL) aims to speed up learning by gradually increasing the difficulty of a task, usually quantified by the achievable expected return. Inspired by the success of CRL in single-agent settings, a few works have attempted to apply CRL to multi-agent reinforcement learning (MARL) using the number of agents to control task difficulty. However, existing works typically use manually defined curricula such as a linear scheme. In this paper, we first apply state-of-the-art single-agent self-paced CRL to sparse reward MARL. Although with satisfying performance, we identify two potential flaws of the curriculum generated by existing reward-based CRL methods: (1) tasks with high returns may not provide informative learning signals and (2) the exacerbated credit assignment difficulty in tasks where more agents yield higher returns. Thereby, we further propose self-paced MARL (SPMARL) to prioritize tasks based on \textit{learning progress} instead of the episode return. Our method not only outperforms baselines in three challenging sparse-reward benchmarks but also converges faster than self-paced CRL.

翻译：课程强化学习（CRL）旨在通过逐步增加任务难度（通常以可实现的期望回报量化）来加速学习过程。受单智能体场景中CRL成功应用的启发，少数研究尝试将CRL应用于多智能体强化学习（MARL），通过控制智能体数量来调节任务难度。然而现有工作通常采用人工定义的课程方案（如线性方案）。本文首先将最先进的单智能体自定进度CRL应用于稀疏奖励MARL。尽管取得了令人满意的性能，但现有基于奖励的CRL方法生成的课程存在两个潜在缺陷：（1）高回报任务可能无法提供有效的学习信号；（2）在更多智能体产生更高回报的任务中，信用分配难度加剧。为此，我们进一步提出自定进度MARL（SPMARL），该方法基于\textit{学习进度}而非回合回报来优先处理任务。我们的方法不仅在三个具有挑战性的稀疏奖励基准测试中优于基线方法，其收敛速度也快于自定进度CRL。