Reinforcement learning (RL) in robotics is often limited by the cost and risk of data collection, motivating experience transfer from a source task to a target task. Offline-to-online RL leverages prior data but typically assumes a given fixed dataset and does not address how to generate reliable data for transfer. We propose World Model-Based Experience Transfer (WOMBET), a framework that jointly generates and utilizes prior data. WOMBET learns a world model in the source task and generates offline data via uncertainty-penalized planning, followed by filtering trajectories with high return and low epistemic uncertainty. It then performs online fine-tuning in the target task using adaptive sampling between offline and online data, enabling a stable transition from prior-driven initialization to task-specific adaptation. We show that the uncertainty-penalized objective provides a lower bound on the true return and derive a finite-sample error decomposition capturing distribution mismatch and approximation error. Empirically, WOMBET improves sample efficiency and final performance over strong baselines on continuous control benchmarks, demonstrating the benefit of jointly optimizing data generation and transfer.

翻译：机器人领域的强化学习常受限于数据收集的成本与风险，这促使研究者从源任务向目标任务进行经验迁移。离线到在线强化学习利用先验数据，但通常假设给定固定数据集，未解决如何生成可靠迁移数据的问题。我们提出基于世界模型的经验迁移框架（WOMBET），该框架联合生成并利用先验数据。WOMBET在源任务中学习世界模型，通过不确定性惩罚规划生成离线数据，随后筛选高回报与低认知不确定性的轨迹。它利用离线和在线数据之间的自适应采样，在目标任务中进行在线微调，从而实现从先验驱动初始化到任务特定适应的稳定过渡。我们证明不确定性惩罚目标为真实回报提供了下界，并推导了捕获分布不匹配与近似误差的有限样本误差分解。实验表明，在连续控制基准测试中，WOMBET相较于强基线方法提升了样本效率与最终性能，验证了联合优化数据生成与迁移的益处。