Designing and analyzing model-based RL (MBRL) algorithms with guaranteed monotonic improvement has been challenging, mainly due to the interdependence between policy optimization and model learning. Existing discrepancy bounds generally ignore the impacts of model shifts, and their corresponding algorithms are prone to degrade performance by drastic model updating. In this work, we first propose a novel and general theoretical scheme for a non-decreasing performance guarantee of MBRL. Our follow-up derived bounds reveal the relationship between model shifts and performance improvement. These discoveries encourage us to formulate a constrained lower-bound optimization problem to permit the monotonicity of MBRL. A further example demonstrates that learning models from a dynamically-varying number of explorations benefit the eventual returns. Motivated by these analyses, we design a simple but effective algorithm CMLO (Constrained Model-shift Lower-bound Optimization), by introducing an event-triggered mechanism that flexibly determines when to update the model. Experiments show that CMLO surpasses other state-of-the-art methods and produces a boost when various policy optimization methods are employed.

翻译：设计和分析具有保证单调改进的基于模型的强化学习（MBRL）算法一直具有挑战性，这主要源于策略优化与模型学习之间的相互依赖性。现有的差异界限通常忽略模型漂移的影响，其对应算法因剧烈模型更新而容易导致性能下降。在本工作中，我们首先提出一种新颖且通用的理论方案，用于确保MBRL的非递减性能保证。我们后续推导的界限揭示了模型漂移与性能改进之间的关系。这些发现促使我们构建一个约束下界优化问题，以允许MBRL的单调性。进一步示例表明，从动态变化的探索次数中学习模型有利于最终回报。受这些分析启发，我们通过引入一种灵活决定何时更新模型的事件触发机制，设计了一个简单但有效的算法CMLO（约束型模型漂移下界优化）。实验表明，CMLO超越了其他最先进的方法，并在采用多种策略优化方法时产生性能提升。