Training reinforcement learning policies using environment interaction data collected from varying policies or dynamics presents a fundamental challenge. Existing works often overlook the distribution discrepancies induced by policy or dynamics shifts, or rely on specialized algorithms with task priors, thus often resulting in suboptimal policy performances and high learning variances. In this paper, we identify a unified strategy for online RL policy learning under diverse settings of policy and dynamics shifts: transition occupancy matching. In light of this, we introduce a surrogate policy learning objective by considering the transition occupancy discrepancies and then cast it into a tractable min-max optimization problem through dual reformulation. Our method, dubbed Occupancy-Matching Policy Optimization (OMPO), features a specialized actor-critic structure equipped with a distribution discriminator and a small-size local buffer. We conduct extensive experiments based on the OpenAI Gym, Meta-World, and Panda Robots environments, encompassing policy shifts under stationary and nonstationary dynamics, as well as domain adaption. The results demonstrate that OMPO outperforms the specialized baselines from different categories in all settings. We also find that OMPO exhibits particularly strong performance when combined with domain randomization, highlighting its potential in RL-based robotics applications

翻译：利用从不同策略或动态环境中收集的交互数据训练强化学习策略，构成了一个基础性挑战。现有研究往往忽视由策略或动态变化引起的分布差异，或依赖具有任务先验知识的专用算法，这通常导致策略性能欠佳且学习方差较高。本文针对策略与动态变化下的在线强化学习策略训练，提出了一种统一策略：转移占用匹配。基于此，我们通过考虑转移占用差异构建了替代策略学习目标，并借助对偶重构将其转化为可处理的极小极大优化问题。我们提出的方法命名为占用匹配策略优化（OMPO），其采用配备分布判别器与小规模局部缓冲区的专用行动者-评论家架构。我们在OpenAI Gym、Meta-World和Panda Robots环境中进行了广泛实验，涵盖静态与非静态动态下的策略变化以及领域自适应场景。实验结果表明，OMPO在所有设定下均优于各类专用基线方法。我们还发现OMPO与领域随机化技术结合时表现出特别优异的性能，这凸显了其在基于强化学习的机器人应用中的潜力。