Experience replay is crucial for off-policy reinforcement learning (RL) methods. By remembering and reusing the experiences from past different policies, experience replay significantly improves the training efficiency and stability of RL algorithms. Many decision-making problems in practice naturally involve multiple agents and require multi-agent reinforcement learning (MARL) under centralized training decentralized execution paradigm. Nevertheless, existing MARL algorithms often adopt standard experience replay where the transitions are uniformly sampled regardless of their importance. Finding prioritized sampling weights that are optimized for MARL experience replay has yet to be explored. To this end, we propose MAC-PO, which formulates optimal prioritized experience replay for multi-agent problems as a regret minimization over the sampling weights of transitions. Such optimization is relaxed and solved using the Lagrangian multiplier approach to obtain the close-form optimal sampling weights. By minimizing the resulting policy regret, we can narrow the gap between the current policy and a nominal optimal policy, thus acquiring an improved prioritization scheme for multi-agent tasks. Our experimental results on Predator-Prey and StarCraft Multi-Agent Challenge environments demonstrate the effectiveness of our method, having a better ability to replay important transitions and outperforming other state-of-the-art baselines.

翻译：经验回放对离策略强化学习方法至关重要。通过记忆并重用来自历史不同策略的经验，经验回放显著提升了强化学习算法的训练效率与稳定性。现实中的诸多决策问题天然涉及多个智能体，需要在集中训练分散执行的范式下进行多智能体强化学习。然而，现有MARL算法通常采用标准经验回放，即所有转移样本被均匀采样，而不考虑其重要性。针对多智能体经验回放优化的优先采样权重问题尚未得到探索。为此，我们提出MAC-PO方法，将多智能体问题的最优优先经验回放构建为关于转移样本采样权重的遗憾最小化问题。通过松弛该优化问题并利用拉格朗日乘子法求解，获得闭式最优采样权重。通过最小化由此产生的策略遗憾，我们能够缩小当前策略与名义最优策略之间的差距，从而为多智能体任务获得更优的优先采样方案。我们在Predator-Prey和StarCraft Multi-Agent Challenge环境上的实验结果证明了该方法的效果，其具备更优的复现重要转移样本的能力，且性能优于其他当前最优基线方法。