Model-based reinforcement learning (MBRL) holds the promise of sample-efficient learning by utilizing a world model, which models how the environment works and typically encompasses components for two tasks: observation modeling and reward modeling. In this paper, through a dedicated empirical investigation, we gain a deeper understanding of the role each task plays in world models and uncover the overlooked potential of sample-efficient MBRL by mitigating the domination of either observation or reward modeling. Our key insight is that while prevalent approaches of explicit MBRL attempt to restore abundant details of the environment via observation models, it is difficult due to the environment's complexity and limited model capacity. On the other hand, reward models, while dominating implicit MBRL and adept at learning compact task-centric dynamics, are inadequate for sample-efficient learning without richer learning signals. Motivated by these insights and discoveries, we propose a simple yet effective approach, HarmonyDream, which automatically adjusts loss coefficients to maintain task harmonization, i.e. a dynamic equilibrium between the two tasks in world model learning. Our experiments show that the base MBRL method equipped with HarmonyDream gains 10%-69% absolute performance boosts on visual robotic tasks and sets a new state-of-the-art result on the Atari 100K benchmark.

翻译：基于模型的强化学习（MBRL）通过构建世界模型来模拟环境运作机制，通常包含观测建模与奖励建模两个任务组件，有望实现样本高效学习。本文通过系统的实证研究，深入揭示了各任务在世界模型中的具体作用，并发现通过缓解观测建模或奖励建模的过度主导问题，能够释放样本高效MBRL中被忽视的潜力。核心洞见在于：显式MBRL的主流方法试图通过观测模型还原环境丰富细节，但受限于环境复杂性与模型容量往往难以达成；而隐性MBRL中占主导的奖励模型虽擅长学习紧凑的任务导向动力学特征，却因缺乏更丰富的学习信号而难以实现样本高效学习。基于上述发现，我们提出简洁有效的HarmonyDream方法，通过自动调整损失系数维持任务调和——即世界模型学习中两个任务间的动态平衡。实验表明，配备HarmonyDream的基础MBRL方法在视觉机器人任务上获得10%-69%的绝对性能提升，并在Atari 100K基准测试中创下新的最优结果。