A significant roadblock to the development of principled multi-agent reinforcement learning is the fact that desired solution concepts like Nash equilibria may be intractable to compute. To overcome this obstacle, we take inspiration from behavioral economics and show that -- by imbuing agents with important features of human decision-making like risk aversion and bounded rationality -- a class of risk-averse quantal response equilibria (RQE) become tractable to compute in all $n$-player matrix and finite-horizon Markov games. In particular, we show that they emerge as the endpoint of no-regret learning in suitably adjusted versions of the games. Crucially, the class of computationally tractable RQE is independent of the underlying game structure and only depends on agents' degree of risk-aversion and bounded rationality. To validate the richness of this class of solution concepts we show that it captures peoples' patterns of play in a number of 2-player matrix games previously studied in experimental economics. Furthermore, we give a first analysis of the sample complexity of computing these equilibria in finite-horizon Markov games when one has access to a generative model and validate our findings on a simple multi-agent reinforcement learning benchmark.

翻译：发展有原则的多智能体强化学习面临的一个主要障碍是，诸如纳什均衡等理想解概念可能难以计算。为克服这一障碍，我们从行为经济学中汲取灵感，并证明——通过赋予智能体风险规避和有限理性等人类决策的重要特征——一类风险规避量化响应均衡（RQE）在所有$n$人矩阵博弈和有限时域马尔可夫博弈中变得易于计算。具体而言，我们证明它们在经过适当调整的博弈版本中，作为无悔学习的终点出现。关键的是，这类计算上易处理的RQE独立于底层博弈结构，仅取决于智能体的风险规避程度和有限理性水平。为验证这类解概念的丰富性，我们证明它捕捉了先前在实验经济学中研究的多个2人矩阵博弈中的人类博弈模式。此外，我们首次分析了在能够访问生成模型的情况下，在有限时域马尔可夫博弈中计算这些均衡的样本复杂度，并在一个简单的多智能体强化学习基准上验证了我们的发现。