Many model-based reinforcement learning (RL) methods follow a similar template: fit a model to previously observed data, and then use data from that model for RL or planning. However, models that achieve better training performance (e.g., lower MSE) are not necessarily better for control: an RL agent may seek out the small fraction of states where an accurate model makes mistakes, or it might act in ways that do not expose the errors of an inaccurate model. As noted in prior work, there is an objective mismatch: models are useful if they yield good policies, but they are trained to maximize their accuracy, rather than the performance of the policies that result from them. In this work, we propose a single objective for jointly training the model and the policy, such that updates to either component increase a lower bound on expected return. To the best of our knowledge, this is the first lower bound for model-based RL that holds globally and can be efficiently estimated in continuous settings; it is the only lower bound that mends the objective mismatch problem. A version of this bound becomes tight under certain assumptions. Optimizing this bound resembles a GAN: a classifier distinguishes between real and fake transitions, the model is updated to produce transitions that look realistic, and the policy is updated to avoid states where the model predictions are unrealistic. Numerical simulations demonstrate that optimizing this bound yields reward maximizing policies and yields dynamics that (perhaps surprisingly) can aid in exploration. We also show that a deep RL algorithm loosely based on our lower bound can achieve performance competitive with prior model-based methods, and better performance on certain hard exploration tasks.

翻译：许多基于模型的强化学习方法遵循相似的模板：将模型拟合到先前观测到的数据，然后利用该模型生成的数据进行强化学习或规划。然而，在训练中表现更优（例如，均方误差更低）的模型并不一定更有利于控制：强化学习智能体可能会寻找准确模型犯错的少数状态，或者采取不暴露不准确模型错误的行为。正如先前工作所指出的，存在一个目标不匹配问题：模型只有在产生良好策略时才有用，但其训练目标是最大化自身的准确性，而非据此生成的策略的性能。在本工作中，我们提出了一个联合训练模型和策略的单一目标，使得对任一组成部分的更新都能增加期望回报的下界。据我们所知，这是首个全局成立且能在连续环境中高效估计的基于模型强化学习下界；也是唯一能够修复目标不匹配问题的下界。该下界在特定假设下可变得紧致。优化该下界类似于生成对抗网络：分类器区分真实与虚假的转移数据，模型更新以产生看起来真实的转移，而策略更新以避开模型预测不真实的状态。数值模拟表明，优化该下界能产生最大化回报的策略，并生成（可能令人意外地）有助于探索的动态。我们还证明，基于我们下界的深度强化学习算法在性能上可与先前的基于模型方法相媲美，并在某些困难探索任务上表现更优。