In order for agents in multi-agent systems (MAS) to be safe, they need to take into account the risks posed by the actions of other agents. However, the dominant paradigm in game theory (GT) assumes that agents are not affected by risk from other agents and only strive to maximise their expected utility. For example, in hybrid human-AI driving systems, it is necessary to limit large deviations in reward resulting from car crashes. Although there are equilibrium concepts in game theory that take into account risk aversion, they either assume that agents are risk-neutral with respect to the uncertainty caused by the actions of other agents, or they are not guaranteed to exist. We introduce a new GT-based Risk-Averse Equilibrium (RAE) that always produces a solution that minimises the potential variance in reward accounting for the strategy of other agents. Theoretically and empirically, we show RAE shares many properties with a Nash Equilibrium (NE), establishing convergence properties and generalising to risk-dominant NE in certain cases. To tackle large-scale problems, we extend RAE to the PSRO multi-agent reinforcement learning (MARL) framework. We empirically demonstrate the minimum reward variance benefits of RAE in matrix games with high-risk outcomes. Results on MARL experiments show RAE generalises to risk-dominant NE in a trust dilemma game and that it reduces instances of crashing by 7x in an autonomous driving setting versus the best performing baseline.

翻译：为确保多智能体系统（MAS）中的智能体安全运行，必须考虑其他智能体行为带来的风险。然而，博弈论（GT）的主流范式假设智能体不受其他智能体风险的干扰，仅致力于最大化自身期望效用。例如，在人机混合自动驾驶系统中，需要限制因车辆碰撞导致的奖励大幅偏差。尽管博弈论中存在考虑风险规避的均衡概念，但这些概念要么假设智能体对其他智能体行为引起的不确定性持风险中性态度，要么无法保证其存在性。本文提出一种基于博弈论的新型风险规避均衡（RAE），该均衡始终能产生最小化潜在奖励方差（考虑其他智能体策略）的解。理论分析与实证研究表明，RAE与纳什均衡（NE）具有诸多相似特性，建立了收敛性保证，并在特定情况下可推广至风险主导纳什均衡。为解决大规模问题，我们将RAE扩展至PSRO多智能体强化学习（MARL）框架。通过高风险结果的矩阵博弈实验，实证展示了RAE在最小化奖励方差方面的优势。MARL实验结果表明：在信任困境博弈中RAE可推广至风险主导纳什均衡，且在自动驾驶场景中，与最佳基线方法相比，碰撞发生率降低7倍。