It is vital to learn effective policies that can be transferred to different domains with dynamics discrepancies in reinforcement learning (RL). In this paper, we consider dynamics adaptation settings where there exists dynamics mismatch between the source domain and the target domain, and one can get access to sufficient source domain data, while can only have limited interactions with the target domain. Existing methods address this problem by learning domain classifiers, performing data filtering from a value discrepancy perspective, etc. Instead, we tackle this challenge from a decoupled representation learning perspective. We perform representation learning only in the target domain and measure the representation deviations on the transitions from the source domain, which we show can be a signal of dynamics mismatch. We also show that representation deviation upper bounds performance difference of a given policy in the source domain and target domain, which motivates us to adopt representation deviation as a reward penalty. The produced representations are not involved in either policy or value function, but only serve as a reward penalizer. We conduct extensive experiments on environments with kinematic and morphology mismatch, and the results show that our method exhibits strong performance on many tasks. Our code is publicly available at https://github.com/dmksjfl/PAR.

翻译：在强化学习（RL）中，学习能够迁移至具有动态差异的不同域的有效策略至关重要。本文考虑动态适应场景，其中源域与目标域之间存在动态失配，且研究者可获取充足的源域数据，但仅能与目标域进行有限交互。现有方法通过训练域分类器、基于价值差异视角进行数据过滤等方式应对此问题。与之不同，我们从解耦表征学习的角度处理这一挑战。我们仅在目标域中进行表征学习，并度量源域转移数据上的表征偏差，我们证明该偏差可作为动态失配的信号。我们还证明表征偏差能够上界给定策略在源域与目标域间的性能差异，这促使我们采用表征偏差作为奖励惩罚项。所生成表征不参与策略或价值函数，仅作为奖励惩罚器使用。我们在具有运动学与形态学失配的环境中进行大量实验，结果表明本方法在多项任务上均表现出强劲性能。代码公开于 https://github.com/dmksjfl/PAR。