The framework of multi-agent learning explores the dynamics of how individual agent strategies evolve in response to the evolving strategies of other agents. Of particular interest is whether or not agent strategies converge to well known solution concepts such as Nash Equilibrium (NE). Most ``fixed order'' learning dynamics restrict an agent's underlying state to be its own strategy. In ``higher order'' learning, agent dynamics can include auxiliary states that can capture phenomena such as path dependencies. We introduce higher-order gradient play dynamics that resemble projected gradient ascent with auxiliary states. The dynamics are ``payoff based'' in that each agent's dynamics depend on its own evolving payoff. While these payoffs depend on the strategies of other agents in a game setting, agent dynamics do not depend explicitly on the nature of the game or the strategies of other agents. In this sense, dynamics are ``uncoupled'' since an agent's dynamics do not depend explicitly on the utility functions of other agents. We first show that for any specific game with an isolated completely mixed-strategy NE, there exist higher-order gradient play dynamics that lead (locally) to that NE, both for the specific game and nearby games with perturbed utility functions. Conversely, we show that for any higher-order gradient play dynamics, there exists a game with a unique isolated completely mixed-strategy NE for which the dynamics do not lead to NE. These results build on prior work that showed that uncoupled fixed-order learning cannot lead to NE in certain instances, whereas higher-order variants can. Finally, we consider the mixed-strategy equilibrium associated with coordination games. While higher-order gradient play can converge to such equilibria, we show such dynamics must be inherently internally unstable.

翻译：多智能体学习框架探讨了个体智能体策略随其他智能体策略演化而变化的动态过程。其中，智能体策略是否收敛于纳什均衡（NE）等经典解概念尤为引人关注。大多数“固定阶”学习动态将智能体的底层状态限制为其自身策略，而“高阶”学习动态可引入辅助状态以捕捉路径依赖等现象。我们提出了高阶梯度博弈动力学，其形式类似于带辅助状态的投影梯度上升。该动力学是“基于收益”的：每个智能体的动态依赖于其自身随时间变化的收益。尽管这些收益在博弈环境中依赖于其他智能体的策略，但智能体动态并不显式依赖于博弈性质或其他智能体的策略。从这一意义而言，该动态是“非耦合”的，因为智能体的动态并不显式依赖于其他智能体的效用函数。我们首先证明：对于任何具有孤立完全混合策略纳什均衡的特定博弈，存在一类高阶梯度博弈动力学，可（局部）收敛至该纳什均衡，不仅适用于该特定博弈，也适用于附近具有扰动效用函数的博弈。反之，我们证明：对于任意高阶梯度博弈动力学，总存在一个具有唯一孤立完全混合策略纳什均衡的博弈，使得该动力学无法收敛至该均衡。这些结果基于先前研究——该研究指出固定阶非耦合学习在某些情形下无法收敛至纳什均衡，而高阶变体可以实现。最后，我们考虑与协调博弈相关的混合策略均衡。尽管高阶梯度博弈可收敛至这类均衡，但分析表明此类动态必然存在内在不稳定性。