We characterize offline data poisoning attacks on Multi-Agent Reinforcement Learning (MARL), where an attacker may change a data set in an attempt to install a (potentially fictitious) unique Markov-perfect Nash equilibrium. We propose the unique Nash set, namely the set of games, specified by their Q functions, with a specific joint policy being the unique Nash equilibrium. The unique Nash set is central to poisoning attacks because the attack is successful if and only if data poisoning pushes all plausible games inside it. The unique Nash set generalizes the reward polytope commonly used in inverse reinforcement learning to MARL. For zero-sum Markov games, both the inverse Nash set and the set of plausible games induced by data are polytopes in the Q function space. We exhibit a linear program to efficiently compute the optimal poisoning attack. Our work sheds light on the structure of data poisoning attacks on offline MARL, a necessary step before one can design more robust MARL algorithms.

翻译：我们刻画了多智能体强化学习（Multi-Agent Reinforcement Learning，MARL）中的离线数据投毒攻击，攻击者可更改数据集，试图植入一个（可能为虚构的）唯一马尔可夫完美纳什均衡。我们提出唯一纳什集，即由其Q函数指定的博弈集合，其中特定的联合策略是唯一的纳什均衡。唯一纳什集是投毒攻击的核心，因为当且仅当数据投毒将所有可能的博弈推入该集合时，攻击成功。唯一纳什集将逆强化学习中常用的奖励多面体推广至MARL。对于零和马尔可夫博弈，逆纳什集和由数据诱导的可能博弈集合均为Q函数空间中的多面体。我们展示了一个线性规划以高效计算最优投毒攻击。我们的工作揭示了离线MARL中数据投毒攻击的结构，这是设计更鲁棒的MARL算法的必要前提。