Autonomous agents deployed in the real world need to be robust against adversarial attacks on sensory inputs. Robustifying agent policies requires anticipating the strongest attacks possible. We demonstrate that existing observation-space attacks on reinforcement learning agents have a common weakness: while effective, their lack of temporal consistency makes them detectable using automated means or human inspection. Detectability is undesirable to adversaries as it may trigger security escalations. We introduce perfect illusory attacks, a novel form of adversarial attack on sequential decision-makers that is both effective and provably statistically undetectable. We then propose the more versatile R-attacks, which result in observation transitions that are consistent with the state-transition function of the adversary-free environment and can be learned end-to-end. Compared to existing attacks, we empirically find R-attacks to be significantly harder to detect with automated methods, and a small study with human subjects suggests they are similarly harder to detect for humans. We propose that undetectability should be a central concern in the study of adversarial attacks on mixed-autonomy settings.

翻译：部署于现实世界的自主智能体需抵御针对感知输入的对抗性攻击。强化智能体策略需要预判可能的最强攻击。我们证明，现有针对强化学习智能体的观测空间攻击存在共同弱点：尽管效果显著，但其缺乏时间一致性，可通过自动化手段或人工审查检测。可检测性对攻击者不利，因其可能触发安全升级。我们提出完美幻觉攻击——一种针对序列决策者的新型对抗攻击形式，兼具有效性与可证明的统计不可检测性。继而提出更具通用性的R攻击，该方法生成的观测转移与无对抗环境的状态转移函数一致，且可通过端到端方式学习。实验表明，与现有攻击相比，R攻击更难被自动化方法检测，而小规模人类受试者研究显示，此类攻击对人类同样难以识别。我们主张，在混合自主环境下的对抗攻击研究中，不可检测性应成为核心关注点。