Training visual reinforcement learning (RL) models in offline datasets is challenging due to overfitting issues in representation learning and overestimation problems in value function. In this paper, we propose a transfer learning method called Collaborative World Models (CoWorld) to improve the performance of visual RL under offline conditions. The core idea is to use an easy-to-interact, off-the-shelf simulator to train an auxiliary RL model as the online ``test bed'' for the offline policy learned in the target domain, which provides a flexible constraint for the value function -- Intuitively, we want to mitigate the overestimation problem of value functions outside the offline data distribution without impeding the exploration of actions with potential advantages. Specifically, CoWorld performs domain-collaborative representation learning to bridge the gap between online and offline hidden state distributions. Furthermore, it performs domain-collaborative behavior learning that enables the source RL agent to provide target-aware value estimation, allowing for effective offline policy regularization. Experiments show that CoWorld significantly outperforms existing methods in offline visual control tasks in DeepMind Control and Meta-World.

翻译：在离线数据集中训练视觉强化学习模型面临表示学习的过拟合问题和价值函数的过估挑战。本文提出名为协作世界模型（CoWorld）的迁移学习方法，旨在改善离线条件下视觉强化学习的性能。核心思想是利用易于交互的现成模拟器训练辅助强化学习模型，作为目标域离线策略的在线"测试平台"，从而为价值函数提供灵活约束——直观而言，我们希望在抑制离线数据分布外价值函数过估问题的同时，不阻碍具有潜在优势的动作探索。具体地，CoWorld通过域协作表示学习弥合在线与离线隐状态分布的差异，并执行域协作行为学习，使源域强化学习智能体提供目标感知的价值估计，实现有效的离线策略正则化。实验表明，在DeepMind Control和Meta-World的离线视觉控制任务中，CoWorld显著优于现有方法。