We study multi-agent reinforcement learning in the setting of episodic Markov decision processes, where multiple agents cooperate via communication through a central server. We propose a provably efficient algorithm based on value iteration that enable asynchronous communication while ensuring the advantage of cooperation with low communication overhead. With linear function approximation, we prove that our algorithm enjoys an $\tilde{\mathcal{O}}(d^{3/2}H^2\sqrt{K})$ regret with $\tilde{\mathcal{O}}(dHM^2)$ communication complexity, where $d$ is the feature dimension, $H$ is the horizon length, $M$ is the total number of agents, and $K$ is the total number of episodes. We also provide a lower bound showing that a minimal $\Omega(dM)$ communication complexity is required to improve the performance through collaboration.

翻译：本文研究在情景马尔可夫决策过程设置下的多智能体强化学习，其中多个智能体通过中央服务器进行通信协作。我们提出一种基于值迭代的可证明高效算法，该算法既能实现异步通信，又能以低通信开销确保协作优势。在线性函数逼近条件下，我们证明该算法可实现 $\tilde{\mathcal{O}}(d^{3/2}H^2\sqrt{K})$ 的遗憾值，通信复杂度为 $\tilde{\mathcal{O}}(dHM^2)$，其中 $d$ 为特征维度，$H$ 为视野长度，$M$ 为智能体总数，$K$ 为总情景数。我们还给出了一个下界，表明必须达到 $\Omega(dM)$ 的最低通信复杂度才能通过协作提升性能。