We present a novel multi-agent RL approach, Selective Multi-Agent Prioritized Experience Relay, in which agents share with other agents a limited number of transitions they observe during training. The intuition behind this is that even a small number of relevant experiences from other agents could help each agent learn. Unlike many other multi-agent RL algorithms, this approach allows for largely decentralized training, requiring only a limited communication channel between agents. We show that our approach outperforms baseline no-sharing decentralized training and state-of-the art multi-agent RL algorithms. Further, sharing only a small number of highly relevant experiences outperforms sharing all experiences between agents, and the performance uplift from selective experience sharing is robust across a range of hyperparameters and DQN variants. A reference implementation of our algorithm is available at https://github.com/mgerstgrasser/super.

翻译：我们提出了一种新颖的多智能体强化学习方法——选择性多智能体优先经验回放。在该方法中，智能体在训练过程中仅与其它智能体共享有限数量的观测转换。其核心直觉在于，即便少量来自其它智能体的相关经验也能帮助每个智能体学习。与许多多智能体强化学习算法不同，该方法支持大规模去中心化训练，仅需智能体间建立有限的通信通道。实验表明，我们的方法优于无需经验共享的基础去中心化训练及当前最先进的多智能体强化学习算法。值得注意的是，仅共享少量高度相关的经验比共享全部经验效果更佳，且选择性经验共享带来的性能提升在多种超参数及DQN变体中均保持稳健。本算法的参考实现可见于https://github.com/mgerstgrasser/super。