Simulated environments are a key piece in the success of Reinforcement Learning (RL), allowing practitioners and researchers to train decision making agents without running expensive experiments on real hardware. Simulators remain a security blind spot, however, enabling adversarial developers to alter the dynamics of their released simulators for malicious purposes. Therefore, in this work we highlight a novel threat, demonstrating how simulator dynamics can be exploited to stealthily implant action-level backdoors into RL agents. The backdoor then allows an adversary to reliably activate targeted actions in an agent upon observing a predefined ``trigger'', leading to potentially dangerous consequences. Traditional backdoor attacks are limited in their strong threat models, assuming the adversary has near full control over an agent's training pipeline, enabling them to both alter and observe agent's rewards. As these assumptions are infeasible to implement within a simulator, we propose a new attack ``Daze'' which is able to reliably and stealthily implant backdoors into RL agents trained for real world tasks without altering or even observing their rewards. We provide formal proof of Daze's effectiveness in guaranteeing attack success across general RL tasks along with extensive empirical evaluations on both discrete and continuous action space domains. We additionally provide the first example of RL backdoor attacks transferring to real, robotic hardware. These developments motivate further research into securing all components of the RL training pipeline to prevent malicious attacks.

翻译：模拟环境是强化学习成功的关键组成部分，它使从业者和研究人员能够训练决策智能体，而无需在真实硬件上运行昂贵的实验。然而，模拟器仍然是一个安全盲点，这使得对抗性开发者能够出于恶意目的改变所发布模拟器的动态。因此，在本工作中，我们强调了一种新的威胁，展示了如何利用模拟器动态将动作级后门隐秘植入强化学习智能体。该后门随后允许攻击者在观察到预定义的“触发器”时可靠地激活智能体中的目标动作，从而导致潜在的危险后果。传统的后门攻击受限于其强大的威胁模型，假设攻击者近乎完全控制智能体的训练流程，能够同时修改和观察智能体的奖励。由于这些假设在模拟器中无法实现，我们提出了一种新的攻击“Daze”，它能够在不修改甚至观察智能体奖励的情况下，将后门可靠且隐秘地植入为现实世界任务训练的强化学习智能体中。我们提供了Daze在保证一般强化学习任务中攻击成功有效性的形式化证明，以及在离散和连续动作空间领域的大量实证评估。此外，我们还首次提出了强化学习后门攻击可转移至真实机器人硬件的示例。这些发展推动了对强化学习训练流程中所有组件进行安全防护的进一步研究，以防止恶意攻击。