Deep reinforcement learning models are vulnerable to adversarial attacks that can decrease a victim's cumulative expected reward by manipulating the victim's observations. Despite the efficiency of previous optimization-based methods for generating adversarial noise in supervised learning, such methods might not be able to achieve the lowest cumulative reward since they do not explore the environmental dynamics in general. In this paper, we provide a framework to better understand the existing methods by reformulating the problem of adversarial attacks on reinforcement learning in the function space. Our reformulation generates an optimal adversary in the function space of the targeted attacks, repelling them via a generic two-stage framework. In the first stage, we train a deceptive policy by hacking the environment, and discover a set of trajectories routing to the lowest reward or the worst-case performance. Next, the adversary misleads the victim to imitate the deceptive policy by perturbing the observations. Compared to existing approaches, we theoretically show that our adversary is stronger under an appropriate noise level. Extensive experiments demonstrate our method's superiority in terms of efficiency and effectiveness, achieving the state-of-the-art performance in both Atari and MuJoCo environments.

翻译：深度强化学习模型容易受到对抗性攻击，攻击者通过操纵受害者的观测来降低其累积期望奖励。尽管先前基于优化的方法在监督学习中能高效生成对抗噪声，但由于未充分探索环境动态，这些方法通常无法实现最低累积奖励。本文通过将强化学习中的对抗性攻击问题重新表述为函数空间中的形式，提供了一个理解现有方法的框架。我们的重新表述能够生成面向攻击目标的函数空间中的最优对抗者，并通过一个通用两阶段框架实现攻击。第一阶段，我们通过劫持环境训练一个欺骗策略，并发现一组导向最低奖励或最差性能的轨迹集合。随后，对抗者通过扰动观测误导受害者模仿该欺骗策略。与现有方法相比，我们理论上证明了在适当噪声水平下，我们的对抗者更具攻击性。大量实验表明，本方法在效率和有效性上均具有优越性，在Atari和MuJoCo环境中均达到了最先进的性能水平。