Mean-field reinforcement learning has become a popular theoretical framework for efficiently approximating large-scale multi-agent reinforcement learning (MARL) problems exhibiting symmetry. However, questions remain regarding the applicability of mean-field approximations: in particular, their approximation accuracy of real-world systems and conditions under which they become computationally tractable. We establish explicit finite-agent bounds for how well the MFG solution approximates the true $N$-player game for two popular mean-field solution concepts. Furthermore, for the first time, we establish explicit lower bounds indicating that MFGs are poor or uninformative at approximating $N$-player games assuming only Lipschitz dynamics and rewards. Finally, we analyze the computational complexity of solving MFGs with only Lipschitz properties and prove that they are in the class of \textsc{PPAD}-complete problems conjectured to be intractable, similar to general sum $N$ player games. Our theoretical results underscore the limitations of MFGs and complement and justify existing work by proving difficulty in the absence of common theoretical assumptions.

翻译：平均场强化学习已成为高效近似大规模对称多智能体强化学习问题的流行理论框架。然而，关于平均场近似的适用性仍存在疑问：特别是对于真实世界系统的近似精度及其计算可行性条件。我们针对两种主流平均场解概念，建立了平均场博弈解对真实$N$玩家博弈近似程度的有穷智能体显式上界。此外，我们首次建立了显式下界，表明仅假设Lipschitz动力学与奖励时，平均场博弈在近似$N$玩家博弈方面表现不佳或缺乏参考价值。最后，我们分析了仅具Lipschitz特性的平均场博弈的计算复杂度，证明其属于部分完整问题类，与一般和$N$玩家博弈类似，被推测为难以求解。我们的理论结果揭示了平均场博弈的局限性，并通过证明缺乏常见理论假设时问题的困难性，对现有工作进行了补充与合理性论证。