We study multi-agent reinforcement learning in the setting of episodic Markov decision processes, where multiple agents cooperate via communication through a central server. We propose a provably efficient algorithm based on value iteration that enable asynchronous communication while ensuring the advantage of cooperation with low communication overhead. With linear function approximation, we prove that our algorithm enjoys an $\tilde{\mathcal{O}}(d^{3/2}H^2\sqrt{K})$ regret with $\tilde{\mathcal{O}}(dHM^2)$ communication complexity, where $d$ is the feature dimension, $H$ is the horizon length, $M$ is the total number of agents, and $K$ is the total number of episodes. We also provide a lower bound showing that a minimal $\Omega(dM)$ communication complexity is required to improve the performance through collaboration.

翻译：我们研究在片段式马尔可夫决策过程环境下的多智能体强化学习问题，其中多个智能体通过中央服务器进行通信协作。我们提出一种基于值迭代的可证明高效算法，该算法在实现异步通信的同时，能以低通信开销确保协作优势。在线性函数近似条件下，我们证明该算法具有$\tilde{\mathcal{O}}(d^{3/2}H^2\sqrt{K})$的遗憾值与$\tilde{\mathcal{O}}(dHM^2)$的通信复杂度，其中$d$为特征维度，$H$为时间跨度长度，$M$为智能体总数，$K$为总片段数。我们还给出了一个下界，表明至少需要$\Omega(dM)$的通信复杂度才能通过协作提升性能。