Model-Based Reinforcement Learning (RL) is widely believed to have the potential to improve sample efficiency by allowing an agent to synthesize large amounts of imagined experience. Experience Replay (ER) can be considered a simple kind of model, which has proved effective at improving the stability and efficiency of deep RL. In principle, a learned parametric model could improve on ER by generalizing from real experience to augment the dataset with additional plausible experience. However, given that learned value functions can also generalize, it is not immediately obvious why model generalization should be better. Here, we provide theoretical and empirical insight into when, and how, we can expect data generated by a learned model to be useful. First, we provide a simple theorem motivating how learning a model as an intermediate step can narrow down the set of possible value functions more than learning a value function directly from data using the Bellman equation. Second, we provide an illustrative example showing empirically how a similar effect occurs in a more concrete setting with neural network function approximation. Finally, we provide extensive experiments showing the benefit of model-based learning for online RL in environments with combinatorial complexity, but factored structure that allows a learned model to generalize. In these experiments, we take care to control for other factors in order to isolate, insofar as possible, the benefit of using experience generated by a learned model relative to ER alone.

翻译：基于模型的强化学习被广泛认为具有通过让智能体合成大量想象经验来提高样本效率的潜力。经验回放可视为一种简单的模型，已被证明能有效提升深度强化学习的稳定性和效率。原则上，学习得到的参数化模型可以通过从真实经验中泛化，用额外的合理经验扩充数据集，从而改进经验回放。然而，由于学习到的值函数也能实现泛化，模型泛化为何更优并非显而易见。本文从理论和实证角度揭示了何时以及如何预期学习模型生成的数据会发挥作用。首先，我们提出一个简洁的定理，证明将模型学习作为中间步骤，比直接使用贝尔曼方程从数据中学习值函数更能缩小可能值函数的集合范围。其次，通过一个示例性案例，实证展示了在神经网络函数逼近的具体场景中如何产生类似效应。最后，我们通过大量实验，证明在具有组合复杂性和可分解结构（允许学习模型进行泛化）的环境中，基于模型的学习对在线强化学习的益处。在这些实验中，我们严格控制其他因素，尽可能分离出学习模型生成的经验相对于纯粹经验回放带来的增益。