Visual model-based RL methods typically encode image observations into low-dimensional representations in a manner that does not eliminate redundant information. This leaves them susceptible to spurious variations -- changes in task-irrelevant components such as background distractors or lighting conditions. In this paper, we propose a visual model-based RL method that learns a latent representation resilient to such spurious variations. Our training objective encourages the representation to be maximally predictive of dynamics and reward, while constraining the information flow from the observation to the latent representation. We demonstrate that this objective significantly bolsters the resilience of visual model-based RL methods to visual distractors, allowing them to operate in dynamic environments. We then show that while the learned encoder is resilient to spirious variations, it is not invariant under significant distribution shift. To address this, we propose a simple reward-free alignment procedure that enables test time adaptation of the encoder. This allows for quick adaptation to widely differing environments without having to relearn the dynamics and policy. Our effort is a step towards making model-based RL a practical and useful tool for dynamic, diverse domains. We show its effectiveness in simulation benchmarks with significant spurious variations as well as a real-world egocentric navigation task with noisy TVs in the background. Videos and code at https://zchuning.github.io/repo-website/.

翻译：基于视觉的模型强化学习方法通常将图像观测编码为低维表示，但这一过程未能消除冗余信息。这使得它们易受虚假变化的影响——即任务无关成分（如背景干扰物或光照条件）的变化。本文提出一种基于视觉的模型强化学习方法，能够学习对这类虚假变化具有鲁棒性的潜在表示。我们的训练目标鼓励该表示最大化对动力学和奖励的预测能力，同时约束从观测到潜在表示的信息流。实验证明，该目标显著增强了基于视觉的模型强化学习方法对视觉干扰物的鲁棒性，使其能够在动态环境中运行。进一步研究表明，尽管学习到的编码器对虚假变化具有鲁棒性，但在显著分布偏移下其并非不变。为解决这一问题，我们提出一种简单的无奖励对齐流程，支持编码器在测试时自适应。这使得编码器能够快速适应差异极大的环境，而无需重新学习动力学和策略。我们的工作旨在推动基于模型的强化学习成为动态、多样化领域中的实用工具。通过在存在显著虚假变化的仿真基准测试以及背景中存在噪点电视的真实世界第一人称导航任务中验证了其有效性。视频和代码见 https://zchuning.github.io/repo-website/。