A backdoor attack allows a malicious user to manipulate the environment or corrupt the training data, thus inserting a backdoor into the trained agent. Such attacks compromise the RL system's reliability, leading to potentially catastrophic results in various key fields. In contrast, relatively limited research has investigated effective defenses against backdoor attacks in RL. This paper proposes the Recovery Triggered States (RTS) method, a novel approach that effectively protects the victim agents from backdoor attacks. RTS involves building a surrogate network to approximate the dynamics model. Developers can then recover the environment from the triggered state to a clean state, thereby preventing attackers from activating backdoors hidden in the agent by presenting the trigger. When training the surrogate to predict states, we incorporate agent action information to reduce the discrepancy between the actions taken by the agent on predicted states and the actions taken on real states. RTS is the first approach to defend against backdoor attacks in a single-agent setting. Our results show that using RTS, the cumulative reward only decreased by 1.41% under the backdoor attack.

翻译：后门攻击允许恶意用户操纵环境或破坏训练数据，从而向受训代理中插入后门。此类攻击会危及RL系统的可靠性，可能导致各个重要领域出现潜在灾难性后果。相比之下，对于防御RL中的后门攻击，相对较少的研究探讨了有效的防御方法。本文提出了一种新颖的方法——恢复触发状态（RTS），有效保护受害代理免受后门攻击。RTS 包括构建替代网络以近似环境动态模型。开发人员可以从触发状态恢复环境到清洁状态，从而防止攻击者通过触发将后门激活在代理中。训练替代模型以预测状态时，我们使用代理的行动信息，以降低代理在预测状态上所采取行动和在真实状态上所采取行动之间的差异。RTS 是第一种在单一代理设置中防御后门攻击的方法。我们的实验结果表明，在后门攻击下，使用 RTS，累计奖励仅下降了 1.41%。