The availability of challenging benchmarks has played a key role in the recent progress of machine learning. In cooperative multi-agent reinforcement learning, the StarCraft Multi-Agent Challenge (SMAC) has become a popular testbed for centralised training with decentralised execution. However, after years of sustained improvement on SMAC, algorithms now achieve near-perfect performance. In this work, we conduct new analysis demonstrating that SMAC lacks the stochasticity and partial observability to require complex *closed-loop* policies. In particular, we show that an *open-loop* policy conditioned only on the timestep can achieve non-trivial win rates for many SMAC scenarios. To address this limitation, we introduce SMACv2, a new version of the benchmark where scenarios are procedurally generated and require agents to generalise to previously unseen settings (from the same distribution) during evaluation. We also introduce the extended partial observability challenge (EPO), which augments SMACv2 to ensure meaningful partial observability. We show that these changes ensure the benchmark requires the use of *closed-loop* policies. We evaluate state-of-the-art algorithms on SMACv2 and show that it presents significant challenges not present in the original benchmark. Our analysis illustrates that SMACv2 addresses the discovered deficiencies of SMAC and can help benchmark the next generation of MARL methods. Videos of training are available at https://sites.google.com/view/smacv2.

翻译：具有挑战性的基准测试在机器学习的最新进展中发挥了关键作用。在合作多智能体强化学习中，星际争霸多智能体挑战（SMAC）已成为集中训练与分散执行的主流测试平台。然而，经过多年的持续改进，算法在SMAC上已实现近乎完美的性能。在本工作中，我们通过新分析表明，SMAC缺乏随机性和部分可观测性，因而不需要复杂的*闭环*策略。具体而言，我们证明仅以时间步为条件的*开环*策略就可在许多SMAC场景中实现非平凡的胜率。为解决这一局限，我们引入SMACv2——基准测试的新版本，其中场景通过程序化生成，要求智能体在评估中泛化至（同一分布下的）未见设置。我们还引入扩展的部分可观测性挑战（EPO），以增强SMACv2确保有意义的局部可观测性。我们证明这些更改确保基准测试需要采用*闭环*策略。我们对SMACv2上的最新算法进行评估，表明其呈现出原始基准测试中不存在的显著挑战。分析表明，SMACv2解决了已发现的SMAC缺陷，有助于对下一代多智能体强化学习方法进行基准测试。训练视频见https://sites.google.com/view/smacv2。