The framework of multi-agent learning explores the dynamics of how individual agent strategies evolve in response to the evolving strategies of other agents. Of particular interest is whether or not agent strategies converge to well known solution concepts such as Nash Equilibrium (NE). Most "fixed order" learning dynamics restrict an agent's underlying state to be its own strategy. In "higher order" learning, agent dynamics can include auxiliary states that can capture phenomena such as path dependencies. We introduce higher-order gradient play dynamics that resemble projected gradient ascent with auxiliary states. The dynamics are "payoff based" in that each agent's dynamics depend on its own evolving payoff. While these payoffs depend on the strategies of other agents in a game setting, agent dynamics do not depend explicitly on the nature of the game or the strategies of other agents. In this sense, dynamics are "uncoupled" since an agent's dynamics do not depend explicitly on the utility functions of other agents. We first show that for any specific game with an isolated completely mixed-strategy NE, there exist higher-order gradient play dynamics that lead (locally) to that NE, both for the specific game and nearby games with perturbed utility functions. Conversely, we show that for any higher-order gradient play dynamics, there exists a game with a unique isolated completely mixed-strategy NE for which the dynamics do not lead to NE. These results build on prior work that showed that uncoupled fixed-order learning cannot lead to NE in certain instances, whereas higher-order variants can. Finally, we consider the mixed-strategy equilibrium associated with coordination games. While higher-order gradient play can converge to such equilibria, we show such dynamics must be inherently internally unstable.

翻译：多智能体学习框架探讨了个体智能体策略如何响应其他智能体策略演变而演变的动态过程。其中特别关注的是智能体策略是否收敛到诸如纳什均衡（NE）等著名解概念。大多数“固定阶”学习动态将智能体的基础状态限制为其自身策略，而“高阶”学习中的智能体动态可包含能够捕捉路径依赖等现象的辅助状态。我们引入了一种高阶梯度博弈动态，其类似于带有辅助状态的投影梯度上升法。该动态是“基于收益”的，因为每个智能体的动态取决于其自身演变的收益。尽管这些收益在博弈环境中依赖于其他智能体的策略，但智能体动态并不显式依赖于博弈的性质或其他智能体的策略。从这个意义上说，动态是“非耦合”的——智能体动态不直接依赖于其他智能体的效用函数。我们首先证明，对于任何具有孤立完全混合策略纳什均衡的特定博弈，存在高阶梯度博弈动态能够（局部地）收敛到该均衡，既适用于该特定博弈，也适用于具有扰动效用函数的邻近博弈。相反地，我们证明对于任何高阶梯度博弈动态，总存在一个具有唯一孤立完全混合策略纳什均衡的博弈，使得该动态无法收敛到该均衡。这些结果基于先前的研究，该研究表明非耦合固定阶学习在某些情况下无法收敛到纳什均衡，而高阶变体则能够实现。最后，我们考虑了与协调博弈相关的混合策略均衡。尽管高阶梯度博弈可以收敛到此类均衡，但我们证明这类动态必须本质上是内部不稳定的。