The framework of multi-agent learning explores the dynamics of how individual agent strategies evolve in response to the evolving strategies of other agents. Of particular interest is whether or not agent strategies converge to well known solution concepts such as Nash Equilibrium (NE). Most ``fixed order'' learning dynamics restrict an agent's underlying state to be its own strategy. In ``higher order'' learning, agent dynamics can include auxiliary states that can capture phenomena such as path dependencies. We introduce higher-order gradient play dynamics that resemble projected gradient ascent with auxiliary states. The dynamics are ``payoff based'' in that each agent's dynamics depend on its own evolving payoff. While these payoffs depend on the strategies of other agents in a game setting, agent dynamics do not depend explicitly on the nature of the game or the strategies of other agents. In this sense, dynamics are ``uncoupled'' since an agent's dynamics do not depend explicitly on the utility functions of other agents. We first show that for any specific game with an isolated completely mixed-strategy NE, there exist higher-order gradient play dynamics that lead (locally) to that NE, both for the specific game and nearby games with perturbed utility functions. Conversely, we show that for any higher-order gradient play dynamics, there exists a game with a unique isolated completely mixed-strategy NE for which the dynamics do not lead to NE. These results build on prior work that showed that uncoupled fixed-order learning cannot lead to NE in certain instances, whereas higher-order variants can. Finally, we consider the mixed-strategy equilibrium associated with coordination games. While higher-order gradient play can converge to such equilibria, we show such dynamics must be inherently internally unstable.

翻译：多智能体学习的研究框架探索了单个智能体策略如何随其他智能体策略的演化而动态调整。其中，智能体策略是否收敛到纳什均衡等经典解概念备受关注。大多数"固定阶"学习动力学将智能体的底层状态限制为其自身策略。而在"高阶"学习中，智能体动力学可包含能捕捉路径依赖等现象的辅助状态。我们提出一类高阶梯度博弈动力学，其形式类似带有辅助状态的投影梯度上升法。这类动力学具有"基于收益"的特点：每个智能体的动力学由其自身动态收益决定。尽管在博弈环境中这些收益取决于其他智能体的策略，但智能体动力学并不显式依赖博弈性质或其他智能体的策略。在此意义上，动力学是"非耦合"的——智能体的动力学并不显式依赖其他智能体的效用函数。我们首先证明：对任何存在孤立完全混合策略纳什均衡的具体博弈，总存在相应的高阶梯度博弈动力学（包括针对该具体博弈及具有扰动效用函数的邻近博弈）能（局部）收敛至该均衡。反之，我们证明：对任意高阶梯度博弈动力学，总能构造一个存在唯一孤立完全混合策略纳什均衡的博弈，使得该动力学无法收敛至该均衡。这些结果建立在先前研究基础上：先前研究表明非耦合固定阶学习在特定情形下无法收敛至纳什均衡，而高阶变体则有可能实现。最后，我们研究了协调博弈中与混合策略均衡相关的收敛性质。虽然高阶梯度博弈可收敛至这类均衡，但我们证明此类动力学必然存在内在不稳定性。