Decentralized cooperative multi-agent deep reinforcement learning (MARL) can be a versatile learning framework, particularly in scenarios where centralized training is either not possible or not practical. One of the key challenges in decentralized deep MARL is the non-stationarity of the learning environment when multiple agents are learning concurrently. A commonly used and efficient scheme for decentralized MARL is independent learning in which agents concurrently update their policies independent of each other. We first show that independent learning does not always converge, while sequential learning where agents update their policies one after another in a sequence is guaranteed to converge to an agent-by-agent optimal solution. In sequential learning, when one agent updates its policy, all other agent's policies are kept fixed, alleviating the challenge of non-stationarity due to concurrent updates in other agents' policies. However, it can be slow because only one agent is learning at any time. Therefore it might also not always be practical. In this work, we propose a decentralized cooperative MARL algorithm based on multi-timescale learning. In multi-timescale learning, all agents learn concurrently, but at different learning rates. In our proposed method, when one agent updates its policy, other agents are allowed to update their policies as well, but at a slower rate. This speeds up sequential learning, while also minimizing non-stationarity caused by other agents updating concurrently. Multi-timescale learning outperforms state-of-the-art decentralized learning methods on a set of challenging multi-agent cooperative tasks in the epymarl (papoudakis2020) benchmark. This can be seen as a first step towards more general decentralized cooperative deep MARL methods based on multi-timescale learning.

翻译：去中心化协作多智能体深度强化学习（MARL）可作为一种通用的学习框架，尤其适用于集中式训练不可行或不切实际的场景。去中心化深度MARL的核心挑战之一是多智能体同时学习时环境存在的非平稳性。独立学习是去中心化MARL中常用且高效的方法，其中智能体独立地同时更新各自策略。我们首先证明独立学习并非总能收敛，而顺序学习（即智能体按序逐个更新策略）则被保证收敛到逐智能体最优解。在顺序学习中，当某个智能体更新策略时，所有其他智能体的策略保持固定，从而缓解了因其他智能体并行更新导致的非平稳性问题。然而，由于每次仅有一个智能体进行学习，该方法可能效率较低，且未必具备实际可行性。为此，本文提出一种基于多时间尺度学习的去中心化协作MARL算法。在多时间尺度学习中，所有智能体以不同学习率并行学习。在提出的方法中，当某个智能体更新其策略时，其他智能体也被允许以较慢速率更新策略。这既加速了顺序学习过程，又最小化了因其他智能体并行更新引发的非平稳性。在epymarl基准测试（papoudakis2020）中一系列具有挑战性的多智能体协作任务上，多时间尺度学习优于当前最先进的去中心化学习方法。该工作可视为迈向基于多时间尺度学习的通用去中心化协作深度MARL方法的第一步。