We introduce networked communication to the mean-field game framework. In particular, we look at oracle-free settings where $N$ decentralised agents learn along a single, non-episodic evolution path of the empirical system, such as we may encounter for a large range of many-agent cooperation problems in the real-world. We provide theoretical evidence that by spreading improved policies through the network in a decentralised fashion, our sample guarantees are upper-bounded by those of the purely independent-learning case. Moreover, we show empirically that our networked method can give faster convergence in practice, while removing the reliance on a centralised controller. We also demonstrate that our decentralised communication architecture brings significant benefits over both the centralised and independent alternatives in terms of robustness and flexibility to unexpected learning failures and changes in population size. For comparison purposes with our new architecture, we modify recent algorithms for the centralised and independent cases to make their practical convergence feasible: while contributing the first empirical demonstrations of these algorithms in our setting of $N$ agents learning along a single system evolution with only local state observability, we additionally display the empirical benefits of our new, networked approach.

翻译：我们首次将网络通信引入均值场博弈框架。特别地，我们关注无预言机设定：N个分散智能体在单一、非周期的经验系统演化路径上共同学习，这类场景广泛存在于现实世界中的多智能体协作问题中。我们提供理论证据表明，通过以分散方式在网络中传播改进策略，样本复杂度保证可被限定在纯独立学习情形之上。此外，我们通过实验证明，所提出的网络化方法在实际应用中能实现更快的收敛，同时消除对中央控制器的依赖。我们还证明，这种分散式通信架构在应对意外学习故障和种群规模变化时，在鲁棒性和灵活性方面均显著优于集中式和独立式方案。为便于与新架构进行对比，我们改进了近期面向集中式和独立式场景的算法，使其在实际应用中实现可行收敛：在仅具备局部状态可观测性的单系统演化路径上实现N个智能体共同学习的设定下，我们不仅首次展示了这些算法的实证结果，更揭示了新提出的网络化方法带来的实际优势。