In this paper, we examine the long-run behavior of regularized, no-regret learning in finite games. A well-known result in the field states that the empirical frequencies of no-regret play converge to the game's set of coarse correlated equilibria; however, our understanding of how the players' actual strategies evolve over time is much more limited - and, in many cases, non-existent. This issue is exacerbated further by a series of recent results showing that only strict Nash equilibria are stable and attracting under regularized learning, thus making the relation between learning and pointwise solution concepts particularly elusive. In lieu of this, we take a more general approach and instead seek to characterize the \emph{setwise} rationality properties of the players' day-to-day play. To that end, we focus on one of the most stringent criteria of setwise strategic stability, namely that any unilateral deviation from the set in question incurs a cost to the deviator - a property known as closedness under better replies (club). In so doing, we obtain a far-reaching equivalence between strategic and dynamic stability: a product of pure strategies is closed under better replies if and only if its span is stable and attracting under regularized learning. In addition, we estimate the rate of convergence to such sets, and we show that methods based on entropic regularization (like the exponential weights algorithm) converge at a geometric rate, while projection-based methods converge within a finite number of iterations, even with bandit, payoff-based feedback.

翻译：本文研究了有限博弈中正则化无遗憾学习的长期行为。该领域的一个著名结论表明，无遗憾博弈的经验频率会收敛到博弈的粗相关均衡集；然而，关于玩家实际策略随时间演化的理解却极为有限——在许多情况下甚至不存在。近期一系列研究结果进一步加剧了这一问题，这些结果表明只有严格纳什均衡在正则化学习下是稳定且吸引的，这使得学习与逐点解概念之间的关系尤为难以捉摸。为此，我们采取更一般化的方法，转而刻画玩家日常博弈的集合理性性质。我们聚焦于集合策略稳定性最严格的标准之一，即任何单方面偏离该集合的行为都会对偏离者产生成本——这一性质被称为"对更优反应的封闭性"。通过这一分析，我们获得了策略稳定性与动态稳定性之间的深远等价关系：纯策略乘积在正则化学习下对更优反应封闭当且仅当其张成空间是稳定且吸引的。此外，我们评估了收敛到此类集合的速率，证明基于熵正则化的方法（如指数权重算法）以几何速率收敛，而基于投影的方法即使在仅知收益的匪徒反馈下也能在有限迭代次数内收敛。