We propose a novel approach to addressing two fundamental challenges in Model-based Reinforcement Learning (MBRL): the computational expense of repeatedly finding a good policy in the learned model, and the objective mismatch between model fitting and policy computation. Our "lazy" method leverages a novel unified objective, Performance Difference via Advantage in Model, to capture the performance difference between the learned policy and expert policy under the true dynamics. This objective demonstrates that optimizing the expected policy advantage in the learned model under an exploration distribution is sufficient for policy computation, resulting in a significant boost in computational efficiency compared to traditional planning methods. Additionally, the unified objective uses a value moment matching term for model fitting, which is aligned with the model's usage during policy computation. We present two no-regret algorithms to optimize the proposed objective, and demonstrate their statistical and computational gains compared to existing MBRL methods through simulated benchmarks.

翻译：我们提出了一种新颖的方法，以应对基于模型的强化学习（MBRL）中两个基本挑战：在学习到的模型中反复寻求良好策略的计算负担，以及模型拟合与策略计算之间的目标不匹配。我们的“懒惰”方法利用了一个新颖的统一目标——“基于模型中的优势性能差”，来捕捉在真实动力学下学习到的策略与专家策略之间的性能差异。该目标表明，在探索分布下优化学习到的模型中的预期策略优势足以用于策略计算，从而相比传统规划方法显著提升了计算效率。此外，统一目标通过使用值矩匹配项进行模型拟合，这与策略计算中的模型使用保持一致。我们提出了两种无憾算法来优化所提出的目标，并通过模拟基准测试展示了它们相比现有MBRL方法的统计和计算优势。