We formulate the problem of \emph{exact unlearning} in reinforcement learning, where the goal is to design an efficient framework that enables the removal of any user's data upon deletion request, i.e., the online learner's output after unlearning is \emph{indistinguishable} from what would have been produced had the deleted user never interacted with the learner. For any $ρ>0$, we show that there exists a reinforcement learning (RL) algorithm that is $ρ$-TV-stable and supports an exact unlearning procedure whose expected computational cost is only a $ρ\sqrt{\ln T}$ fraction of the computational cost of retraining from scratch. We construct such a $ρ$-TV-stable RL algorithm for tabular Markov decision processes (MDPs), which achieves a regret bound of $\mathcal{O}(H^2 \sqrt{SAT} + H^3 S^2 A + {H^{2.5} S^2 A}/ρ)$, where $S, A, H$, and $T$ denote the number of states, the number of actions, the episode horizon, and the number of episodes, respectively. We also establish a lower bound of $Ω(H\sqrt{\!SAT}\! +\! {SAH}/ρ)$ for $ρ$-TV-stable RL algorithms, showing that our algorithm is nearly minimax optimal.

翻译：我们提出了强化学习中“精确遗忘”问题，目标在于设计一个高效框架，使得在收到用户删除请求后能够移除其数据。具体而言，遗忘处理后在线学习器的输出与从未与学习器交互过的用户所产生的输出是“不可区分的”。对于任意ρ>0，我们证明存在一个ρ-TV稳定的强化学习算法，其支持精确遗忘过程，预期计算成本仅为从头再训练计算成本的一个分数ρ√lnT。我们为表格型马尔可夫决策过程构造了这样一个ρ-TV稳定的强化学习算法，其遗憾界为O(H²√(SAT)+H³S²A+H²·⁵S²A/ρ)，其中S、A、H和T分别表示状态数、动作数、回合周期和回合数。我们还为ρ-TV稳定的强化学习算法建立了下界Ω(H√(SAT)+SAH/ρ)，表明我们的算法几乎达到了极小化最优。