Beyond specific settings, many multi-agent learning algorithms fail to converge to an equilibrium solution, instead displaying complex, non-stationary behaviours such as recurrent or chaotic orbits. In fact, recent literature suggests that such complex behaviours are likely to occur when the number of agents increases. In this paper, we study Q-learning dynamics in network polymatrix normal-form games where the network structure is drawn from classical random graph models. In particular, we focus on the Erdős-Rényi model, which is used to analyze connectivity in distributed systems, and the Stochastic Block model, which generalizes the above by accounting for community structures that naturally arise in multi-agent systems. In each setting, we establish sufficient conditions under which the agents' joint strategies converge to a unique equilibrium. We investigate how this condition depends on the exploration rates, payoff matrices and, crucially, the probabilities of interaction between network agents. We validate our theoretical findings through numerical simulations and demonstrate that convergence can be reliably achieved in many-agent systems, provided interactions in the network are controlled.

翻译：在特定设定之外，许多多智能体学习算法无法收敛至均衡解，反而展现出复杂、非平稳的行为模式，如周期性或混沌轨道。事实上，近期文献表明，当智能体数量增加时，此类复杂行为很可能发生。本文研究网络多矩阵正规形式博弈中的Q学习动力学，其中网络结构取自经典随机图模型。我们特别关注用于分析分布式系统连通性的Erdős-Rényi模型，以及通过考虑多智能体系统中自然出现的社区结构而推广该模型的随机分块模型。在每种设定下，我们建立了智能体联合策略收敛至唯一均衡的充分条件。我们研究了该条件如何依赖于探索率、收益矩阵以及网络智能体间交互概率这一关键因素。通过数值模拟验证了理论发现，并证明只要控制网络中的交互，多智能体系统能够可靠实现收敛。