When Reinforcement Learning (RL) agents are deployed in practice, they might impact their environment and change its dynamics. We propose a new framework to model this phenomenon, where the current environment depends on the deployed policy as well as its previous dynamics. This is a generalization of Performative RL (PRL) [Mandal et al., 2023]. Unlike PRL, our framework allows to model scenarios where the environment gradually adjusts to a deployed policy. We adapt two algorithms from the performative prediction literature to our setting and propose a novel algorithm called Mixed Delayed Repeated Retraining (MDRR). We provide conditions under which these algorithms converge and compare them using three metrics: number of retrainings, approximation guarantee, and number of samples per deployment. MDRR is the first algorithm in this setting which combines samples from multiple deployments in its training. This makes MDRR particularly suitable for scenarios where the environment's response strongly depends on its previous dynamics, which are common in practice. We experimentally compare the algorithms using a simulation-based testbed and our results show that MDRR converges significantly faster than previous approaches.

翻译：当强化学习（RL）智能体在实际环境中部署时，其行为可能影响并改变环境动态。本文提出一种新的建模框架来描述这一现象，其中当前环境状态取决于已部署策略及其历史动态。该框架是执行性强化学习（PRL）[Mandal et al., 2023] 的泛化形式。与PRL不同，本框架能够对环境逐步适应已部署策略的场景进行建模。我们将执行性预测领域的两种算法适配至本设定，并提出名为混合延迟重复训练（MDRR）的新算法。我们给出了这些算法的收敛条件，并通过三项指标进行对比分析：重复训练次数、近似保证程度以及单次部署的样本需求量。MDRR是该领域首个在训练中融合多轮部署样本的算法，这使其特别适用于环境响应强烈依赖历史动态的实践场景。我们通过基于仿真的测试平台进行实验对比，结果表明MDRR的收敛速度显著优于现有方法。