Dyna-style model-based reinforcement learning contains two phases: model rollouts to generate sample for policy learning and real environment exploration using current policy for dynamics model learning. However, due to the complex real-world environment, it is inevitable to learn an imperfect dynamics model with model prediction error, which can further mislead policy learning and result in sub-optimal solutions. In this paper, we propose $\texttt{COPlanner}$, a planning-driven framework for model-based methods to address the inaccurately learned dynamics model problem with conservative model rollouts and optimistic environment exploration. $\texttt{COPlanner}$ leverages an uncertainty-aware policy-guided model predictive control (UP-MPC) component to plan for multi-step uncertainty estimation. This estimated uncertainty then serves as a penalty during model rollouts and as a bonus during real environment exploration respectively, to choose actions. Consequently, $\texttt{COPlanner}$ can avoid model uncertain regions through conservative model rollouts, thereby alleviating the influence of model error. Simultaneously, it explores high-reward model uncertain regions to reduce model error actively through optimistic real environment exploration. $\texttt{COPlanner}$ is a plug-and-play framework that can be applied to any dyna-style model-based methods. Experimental results on a series of proprioceptive and visual continuous control tasks demonstrate that both sample efficiency and asymptotic performance of strong model-based methods are significantly improved combined with $\texttt{COPlanner}$.

翻译：Dyna式基于模型的强化学习包含两个阶段：模型展开用于生成策略学习的样本，以及利用当前策略进行真实环境探索以学习动力学模型。然而，由于真实环境的复杂性，不可避免地会学习到存在模型预测误差的不完美动力学模型，这进而可能误导策略学习，导致次优解。本文提出$\texttt{COPlanner}$，一种面向基于模型方法的规划驱动框架，通过保守模型展开与乐观环境探索，解决动力学模型学习不准确的问题。$\texttt{COPlanner}$利用基于不确定性感知的策略引导模型预测控制（UP-MPC）组件，规划多步不确定性估计。该估计的不确定性分别在模型展开中作为惩罚项、在真实环境探索中作为奖励项，用于动作选择。因此，$\texttt{COPlanner}$通过保守模型展开避免模型不确定区域，从而缓解模型误差的影响；同时，通过乐观真实环境探索主动探索高奖励模型不确定区域，以降低模型误差。$\texttt{COPlanner}$是一种即插即用框架，可应用于任意Dyna式基于模型的方法。在一系列本体感知与视觉连续控制任务上的实验结果表明，结合$\texttt{COPlanner}$后，强基于模型方法的样本效率与渐近性能均得到显著提升。