Evaluating the worst-case performance of a reinforcement learning (RL) agent under the strongest/optimal adversarial perturbations on state observations (within some constraints) is crucial for understanding the robustness of RL agents. However, finding the optimal adversary is challenging, in terms of both whether we can find the optimal attack and how efficiently we can find it. Existing works on adversarial RL either use heuristics-based methods that may not find the strongest adversary, or directly train an RL-based adversary by treating the agent as a part of the environment, which can find the optimal adversary but may become intractable in a large state space. This paper introduces a novel attacking method to find the optimal attacks through collaboration between a designed function named "actor" and an RL-based learner named "director". The actor crafts state perturbations for a given policy perturbation direction, and the director learns to propose the best policy perturbation directions. Our proposed algorithm, PA-AD, is theoretically optimal and significantly more efficient than prior RL-based works in environments with large state spaces. Empirical results show that our proposed PA-AD universally outperforms state-of-the-art attacking methods in various Atari and MuJoCo environments. By applying PA-AD to adversarial training, we achieve state-of-the-art empirical robustness in multiple tasks under strong adversaries. The codebase is released at https://github.com/umd-huang-lab/paad_adv_rl.

翻译：评估强化学习（RL）智能体在状态观测上受到最强/最优对抗扰动（在特定约束下）时的最差性能，对于理解RL智能体的鲁棒性至关重要。然而，寻找最优对手既具挑战性，又涉及能否找到最优攻击以及找到它的效率问题。现有对抗RL研究要么采用基于启发式的方法（可能无法找到最强对手），要么通过将智能体视为环境的一部分直接训练基于RL的对手（虽能找到最优对手但在大状态空间中可能难以处理）。本文提出一种新颖的攻击方法，通过设计名为“actor”的函数与基于RL的学习器“director”协作来寻找最优攻击。Actor针对给定策略扰动方向构造状态扰动，而Director学习提出最佳策略扰动方向。我们提出的算法PA-AD在理论上具有最优性，并在大状态空间环境中显著优于先前基于RL的方法。实验结果表明，我们提出的PA-AD在各种Atari和MuJoCo环境中普遍优于最先进的攻击方法。通过将PA-AD应用于对抗训练，我们在多个任务中实现了强对抗者下的最先进经验鲁棒性。代码库已发布在https://github.com/umd-huang-lab/paad_adv_rl。