We study multi-agent reinforcement learning in the setting of episodic Markov decision processes, where multiple agents cooperate via communication through a central server. We propose a provably efficient algorithm based on value iteration that enable asynchronous communication while ensuring the advantage of cooperation with low communication overhead. With linear function approximation, we prove that our algorithm enjoys an $\tilde{\mathcal{O}}(d^{3/2}H^2\sqrt{K})$ regret with $\tilde{\mathcal{O}}(dHM^2)$ communication complexity, where $d$ is the feature dimension, $H$ is the horizon length, $M$ is the total number of agents, and $K$ is the total number of episodes. We also provide a lower bound showing that a minimal $\Omega(dM)$ communication complexity is required to improve the performance through collaboration.

翻译：我们研究在片段式马尔可夫决策过程中的多智能体强化学习问题，其中多个智能体通过中央服务器进行通信协作。我们提出一种基于值迭代的可证明高效算法，该算法在实现异步通信的同时，以低通信开销确保协作优势。在线性函数逼近条件下，我们证明该算法的遗憾界为$\tilde{\mathcal{O}}(d^{3/2}H^2\sqrt{K})$，通信复杂度为$\tilde{\mathcal{O}}(dHM^2)$，其中$d$为特征维度，$H$为片段长度，$M$为智能体总数，$K$为总片段数。我们还给出了下界，表明通过协作提升性能至少需要$\Omega(dM)$的通信复杂度。