Experience replay is crucial for off-policy reinforcement learning (RL) methods. By remembering and reusing the experiences from past different policies, experience replay significantly improves the training efficiency and stability of RL algorithms. Many decision-making problems in practice naturally involve multiple agents and require multi-agent reinforcement learning (MARL) under centralized training decentralized execution paradigm. Nevertheless, existing MARL algorithms often adopt standard experience replay where the transitions are uniformly sampled regardless of their importance. Finding prioritized sampling weights that are optimized for MARL experience replay has yet to be explored. To this end, we propose \name, which formulates optimal prioritized experience replay for multi-agent problems as a regret minimization over the sampling weights of transitions. Such optimization is relaxed and solved using the Lagrangian multiplier approach to obtain the close-form optimal sampling weights. By minimizing the resulting policy regret, we can narrow the gap between the current policy and a nominal optimal policy, thus acquiring an improved prioritization scheme for multi-agent tasks. Our experimental results on Predator-Prey and StarCraft Multi-Agent Challenge environments demonstrate the effectiveness of our method, having a better ability to replay important transitions and outperforming other state-of-the-art baselines.

翻译：经验回放对于离策略强化学习方法至关重要。通过记忆并复用来自历史不同策略的经验，经验回放显著提升了强化学习算法的训练效率与稳定性。许多实际决策问题天然涉及多个智能体，需要在集中训练分布式执行范式下进行多智能体强化学习。然而，现有MARL算法通常采用标准经验回放机制，即以均匀概率采样所有经验转换而忽略其重要程度。针对多智能体经验回放优化优先采样权重的问题尚未得到探索。为此，我们提出\name方法，将多智能体问题的最优优先经验回放构建为基于经验转换采样权重的遗憾最小化问题。通过拉格朗日乘子法对该优化问题进行松弛求解，获得闭合形式的最优采样权重。通过最小化产生的策略遗憾，我们缩小当前策略与名义最优策略之间的差距，从而获得针对多智能体任务的改进优先级方案。在Predator-Prey和星际争霸多智能体挑战环境上的实验结果表明，我们的方法具有更优的转换重要性回放能力，性能超越其他先进基线方法。