Although the field of multi-agent reinforcement learning (MARL) has made considerable progress in the last years, solving systems with a large number of agents remains a hard challenge. Graphon mean field games (GMFGs) enable the scalable analysis of MARL problems that are otherwise intractable. By the mathematical structure of graphons, this approach is limited to dense graphs which are insufficient to describe many real-world networks such as power law graphs. Our paper introduces a novel formulation of GMFGs, called LPGMFGs, which leverages the graph theoretical concept of $L^p$ graphons and provides a machine learning tool to efficiently and accurately approximate solutions for sparse network problems. This especially includes power law networks which are empirically observed in various application areas and cannot be captured by standard graphons. We derive theoretical existence and convergence guarantees and give empirical examples that demonstrate the accuracy of our learning approach for systems with many agents. Furthermore, we extend the Online Mirror Descent (OMD) learning algorithm to our setup to accelerate learning speed, empirically show its capabilities, and conduct a theoretical analysis using the novel concept of smoothed step graphons. In general, we provide a scalable, mathematically well-founded machine learning approach to a large class of otherwise intractable problems of great relevance in numerous research fields.

翻译：尽管多智能体强化学习（MARL）领域近年来取得了显著进展，但解决包含大量智能体的系统仍然是一个艰巨的挑战。图平均场博弈（GMFGs）能够对原本难以处理的MARL问题进行可扩展分析。然而，受图论数学结构的限制，该方法仅适用于稠密图，而无法描述许多现实世界中的网络（如幂律图）。本文提出了一种名为LPGMFGs的GMFGs新形式，该形式利用了$L^p$图论的图论概念，并提供了一种机器学习工具，能够高效且精确地逼近稀疏网络问题的解。这尤其包括幂律网络——此类网络已在多个应用领域得到实证观察，但无法被标准图论所刻画。我们推导了理论上的存在性和收敛性保证，并通过实证示例展示了所提学习方法在包含大量智能体系统中的准确性。此外，我们将在线镜像下降（OMD）学习算法扩展至我们的设置中以加速学习速度，通过实验展示了其能力，并利用平滑步进图论的新概念进行了理论分析。总的来说，我们为一大批在众多研究领域中具有重大意义但原本难以处理的问题提供了一种可扩展、数学基础扎实的机器学习方法。