This paper investigates policy resilience to training-environment poisoning attacks on reinforcement learning (RL) policies, with the goal of recovering the deployment performance of a poisoned RL policy. Due to the fact that the policy resilience is an add-on concern to RL algorithms, it should be resource-efficient, time-conserving, and widely applicable without compromising the performance of RL algorithms. This paper proposes such a policy-resilience mechanism based on an idea of knowledge sharing. We summarize the policy resilience as three stages: preparation, diagnosis, recovery. Specifically, we design the mechanism as a federated architecture coupled with a meta-learning manner, pursuing an efficient extraction and sharing of the environment knowledge. With the shared knowledge, a poisoned agent can quickly identify the deployment condition and accordingly recover its policy performance. We empirically evaluate the resilience mechanism for both model-based and model-free RL algorithms, showing its effectiveness and efficiency in restoring the deployment performance of a poisoned policy.

翻译：本文研究强化学习策略在训练环境投毒攻击下的策略韧性，旨在恢复已受投毒攻击策略的部署性能。考虑到策略韧性是强化学习算法的附加关注点，该机制应具备资源高效性、时间节约性及广泛适用性，且不损害强化学习算法性能。本文基于知识共享理念提出一种策略韧性机制，将策略韧性归纳为三个阶段：准备阶段、诊断阶段与恢复阶段。具体而言，我们采用联邦架构与元学习范式相结合的方式设计该机制，以实现环境知识的高效提取与共享。借助共享知识，受攻击智能体能够快速识别部署环境状态，并据此恢复策略性能。我们通过基于模型与无模型两类强化学习算法对韧性机制进行实证评估，结果表明该机制在恢复受毒攻策略部署性能方面具有有效性与高效性。