The framework of multi-agent learning explores the dynamics of how individual agent strategies evolve in response to the evolving strategies of other agents. Of particular interest is whether or not agent strategies converge to well known solution concepts such as Nash Equilibrium (NE). Most "fixed order" learning dynamics restrict an agent's underlying state to be its own strategy. In "higher order" learning, agent dynamics can include auxiliary states that can capture phenomena such as path dependencies. We introduce higher-order gradient play dynamics that resemble projected gradient ascent with auxiliary states. The dynamics are "payoff based" in that each agent's dynamics depend on its own evolving payoff. While these payoffs depend on the strategies of other agents in a game setting, agent dynamics do not depend explicitly on the nature of the game or the strategies of other agents. In this sense, dynamics are "uncoupled" since an agent's dynamics do not depend explicitly on the utility functions of other agents. We first show that for any specific game with an isolated completely mixed-strategy NE, there exist higher-order gradient play dynamics that lead (locally) to that NE, both for the specific game and nearby games with perturbed utility functions. Conversely, we show that for any higher-order gradient play dynamics, there exists a game with a unique isolated completely mixed-strategy NE for which the dynamics do not lead to NE. These results build on prior work that showed that uncoupled fixed-order learning cannot lead to NE in certain instances, whereas higher-order variants can. Finally, we consider the mixed-strategy equilibrium associated with coordination games. While higher-order gradient play can converge to such equilibria, we show such dynamics must be inherently internally unstable.

翻译：多智能体学习框架探讨了个体智能体策略如何随着其他智能体策略的演变而变化的动态过程。其中特别关注的是智能体策略是否收敛于诸如纳什均衡（NE）等公认的解概念。大多数“固定阶”学习动态将智能体的底层状态限制为其自身的策略。而在“高阶”学习中，智能体的动态可以包含能捕捉路径依赖等现象的辅助状态。我们引入了高阶梯度博弈动态，其形式类似于带有辅助状态的投影梯度上升。这些动态是“基于收益”的，因为每个智能体的动态取决于其自身不断变化的收益。尽管这些收益在博弈环境中依赖于其他智能体的策略，但智能体的动态并不显式依赖于博弈的性质或其他智能体的策略。在此意义上，这些动态是“非耦合”的，因为智能体的动态不显式依赖于其他智能体的效用函数。我们首先证明，对于任意具有孤立完全混合策略纳什均衡的具体博弈，存在高阶梯度博弈动态（局部）收敛到此均衡，这既适用于该具体博弈，也适用于具有扰动效用函数的邻近博弈。相反地，我们证明对于任意高阶梯度博弈动态，存在一个具有唯一孤立完全混合策略纳什均衡的博弈，使得该动态不收敛于此均衡。这些结果建立在先前工作的基础上，该工作表明非耦合固定阶学习在某些情况下无法收敛到纳什均衡，而高阶变体则能实现。最后，我们考虑与协调博弈相关的混合策略均衡。尽管高阶梯度博弈动态能够收敛到这类均衡，但我们证明此类动态必然存在内在的不稳定性。