Simulated environments are a key piece in the success of Reinforcement Learning (RL), allowing practitioners and researchers to train decision making agents without running expensive experiments on real hardware. Simulators remain a security blind spot, however, enabling adversarial developers to alter the dynamics of their released simulators for malicious purposes. Therefore, in this work we highlight a novel threat, demonstrating how simulator dynamics can be exploited to stealthily implant action-level backdoors into RL agents. The backdoor then allows an adversary to reliably activate targeted actions in an agent upon observing a predefined ``trigger'', leading to potentially dangerous consequences. Traditional backdoor attacks are limited in their strong threat models, assuming the adversary has near full control over an agent's training pipeline, enabling them to both alter and observe agent's rewards. As these assumptions are infeasible to implement within a simulator, we propose a new attack ``Daze'' which is able to reliably and stealthily implant backdoors into RL agents trained for real world tasks without altering or even observing their rewards. We provide formal proof of Daze's effectiveness in guaranteeing attack success across general RL tasks along with extensive empirical evaluations on both discrete and continuous action space domains. We additionally provide the first example of RL backdoor attacks transferring to real, robotic hardware. These developments motivate further research into securing all components of the RL training pipeline to prevent malicious attacks.

翻译：模拟环境是强化学习（RL）成功的关键组成部分，使从业者和研究人员能够在无需对真实硬件进行昂贵实验的情况下训练决策智能体。然而，模拟器仍是一个安全盲区，允许恶意开发者通过修改其发布的模拟器动态来实现有害目的。因此，在本工作中，我们揭示了一种新型威胁，展示了如何利用模拟器动态向RL智能体中隐秘植入动作级后门。该后门使得攻击者能够在观测到预定义的“触发器”时，可靠地激活智能体中的目标动作，从而可能导致危险后果。传统后门攻击受限于其强威胁模型，假设攻击者几乎完全控制智能体的训练流程，既能修改也能观测智能体的奖励。由于这些假设在模拟器场景中难以实现，我们提出了一种新型攻击方法“Daze”，该方法能够在无需修改甚至无需观测奖励的情况下，可靠且隐蔽地向面向真实世界任务训练的RL智能体中植入后门。我们通过形式化证明验证了Daze在通用RL任务中保证攻击成功率的有效性，并在离散与连续动作空间领域进行了广泛的实证评估。此外，我们首次展示了RL后门攻击可迁移至真实机器人硬件。这些进展推动了对RL训练全流程组件安全防护的进一步研究，以防范恶意攻击。