We study off-dynamics offline reinforcement learning, where the goal is to learn a policy from offline source and limited target datasets with mismatched dynamics. Existing methods either penalize the reward or discard source transitions occurring in parts of the transition space with high dynamics shift. As a result, they optimize the policy using data from low-shift regions, limiting exploration of high-reward states in the target domain that do not fall within these regions. Consequently, such methods often fail when the dynamics shift is significant or the optimal trajectories lie outside the low-shift regions. To overcome this limitation, we propose MOBODY, a Model-Based Off-Dynamics Offline RL algorithm that optimizes a policy using learned target dynamics transitions to explore the target domain, rather than only being trained with the low dynamics-shift transitions. For the dynamics learning, built on the observation that achieving the same next state requires taking different actions in different domains, MOBODY employs separate action encoders for each domain to encode different actions to the shared latent space while sharing a unified representation of states and a common transition function. We further introduce a target Q-weighted behavior cloning loss in policy optimization to avoid out-of-distribution actions, which push the policy toward actions with high target-domain Q-values, rather than high source domain Q-values or uniformly imitating all actions in the offline dataset. We evaluate MOBODY on a wide range of MuJoCo and Adroit benchmarks, demonstrating that it outperforms state-of-the-art off-dynamics RL baselines as well as policy learning methods based on different dynamics learning baselines, with especially pronounced improvements in challenging scenarios where existing methods struggle.

翻译：本文研究异动态离线强化学习，其目标是从动态特性不匹配的离线源数据集和有限目标数据集中学习策略。现有方法要么对奖励进行惩罚，要么丢弃发生在动态偏移较大的转移空间部分的源转移数据。因此，这些方法仅利用低偏移区域的数据优化策略，限制了对目标域中不位于这些区域内的高奖励状态的探索。当动态偏移显著或最优轨迹位于低偏移区域之外时，此类方法通常会失效。为克服这一局限，我们提出MOBODY——一种基于模型的异动态离线强化学习算法，该算法利用学习到的目标动态转移来优化策略以探索目标域，而非仅使用低动态偏移的转移数据进行训练。在动态学习方面，基于"达到相同下一状态需要在不同域中采取不同行动"这一观察，MOBODY为每个域配备独立的行动编码器，将不同行动编码到共享潜在空间，同时共享统一的状态表示和通用的转移函数。我们进一步在策略优化中引入目标Q加权行为克隆损失，以避免分布外行动，该损失推动策略趋向具有高目标域Q值的行动，而非高源域Q值的行动或均匀模仿离线数据集中的所有行动。我们在多种MuJoCo和Adroit基准测试中评估MOBODY，结果表明其性能优于最先进的异动态强化学习基线方法以及基于不同动态学习基线的策略学习方法，在现有方法难以应对的挑战性场景中改进尤为显著。