Multi-Agent Experience Replay (MER) is a key component of off-policy reinforcement learning~(RL) algorithms. By remembering and reusing experiences from the past, experience replay significantly improves the stability of RL algorithms and their learning efficiency. In many scenarios, multiple agents interact in a shared environment during online training under centralized training and decentralized execution~(CTDE) paradigm. Current multi-agent reinforcement learning~(MARL) algorithms consider experience replay with uniform sampling or based on priority weights to improve transition data sample efficiency in the sampling phase. However, moving transition data histories for each agent through the processor memory hierarchy is a performance limiter. Also, as the agents' transitions continuously renew every iteration, the finite cache capacity results in increased cache misses. To this end, we propose \name, that repeatedly reuses the transitions~(experiences) for a window of $n$ steps in order to improve the cache locality and minimize the transition data movement, instead of sampling new transitions at each step. Specifically, our optimization uses priority weights to select the transitions so that only high-priority transitions will be reused frequently, thereby improving the cache performance. Our experimental results on the Predator-Prey environment demonstrate the effectiveness of reusing the essential transitions based on the priority weights, where we observe an end-to-end training time reduction of $25.4\%$~(for $32$ agents) compared to existing prioritized MER algorithms without notable degradation in the mean reward.

翻译：摘要：多智能体经验回放（Multi-Agent Experience Replay, MER）是非策略强化学习算法中的关键组成部分。通过记忆并复用过往经验，经验回放可显著提升强化学习算法的稳定性及其学习效率。在集中式训练与分布式执行（CTDE）范式下，多个智能体在在线训练过程中于共享环境中进行交互。当前的多智能体强化学习算法在采样阶段多采用均匀采样或基于优先权重的经验回放机制，以提升转换数据的采样效率。然而，在处理器存储层次结构中移动每个智能体的历史转换数据已成为性能瓶颈。此外，由于智能体的转换数据每轮迭代持续更新，有限的缓存容量会引发更高的缓存未命中率。为此，我们提出AccMER方法——该方法通过在n步时间窗口内反复复用转换数据（经验），优先提升缓存局部性并最小化转换数据迁移量，而非在每个时间步采样全新转换数据。具体而言，我们采用优先权重筛选转换数据，使得仅高频次复用高优先级的转换数据，从而优化缓存性能。在Predator-Prey环境中的实验结果表明，基于优先权重复用关键转换数据的策略具有有效性：与现有带优先化的MER算法相比，本方法在32个智能体场景下实现端到端训练时间缩减25.4%，且平均奖励无明显降级。