While reinforcement learning (RL) methods that learn an internal model of the environment have the potential to be more sample efficient than their model-free counterparts, learning to model raw observations from high dimensional sensors can be challenging. Prior work has addressed this challenge by learning low-dimensional representation of observations through auxiliary objectives, such as reconstruction or value prediction. However, the alignment between these auxiliary objectives and the RL objective is often unclear. In this work, we propose a single objective which jointly optimizes a latent-space model and policy to achieve high returns while remaining self-consistent. This objective is a lower bound on expected returns. Unlike prior bounds for model-based RL on policy exploration or model guarantees, our bound is directly on the overall RL objective. We demonstrate that the resulting algorithm matches or improves the sample-efficiency of the best prior model-based and model-free RL methods. While sample efficient methods typically are computationally demanding, our method attains the performance of SAC in about 50% less wall-clock time.

翻译：尽管通过学习环境内部模型的强化学习方法在样本效率上具有超越无模型方法的潜力，但学习对高维传感器原始观测进行建模仍颇具挑战。先前研究通过辅助目标（如重构或价值预测）学习低维观测表征来应对这一挑战。然而，这些辅助目标与强化学习目标之间的对齐关系往往不明确。本文提出一个单一目标，该目标联合优化隐空间模型与策略，在保持自洽性的同时实现高回报。该目标为期望回报的下界。不同于先前基于模型强化学习中针对策略探索或模型保证的下界，我们的下界直接作用于整体强化学习目标。我们证明，由此产生的算法在样本效率上达到或超越先前最优的基于模型与无模型强化学习方法。尽管样本高效方法通常计算密集，但我们的方法可在减少约50%的墙钟时间条件下实现SAC算法的性能。