A key component of model-based reinforcement learning (RL) is a dynamics model that predicts the outcomes of actions. Errors in this predictive model can degrade the performance of model-based controllers, and complex Markov decision processes (MDPs) can present exceptionally difficult prediction problems. To mitigate this issue, we propose predictable MDP abstraction (PMA): instead of training a predictive model on the original MDP, we train a model on a transformed MDP with a learned action space that only permits predictable, easy-to-model actions, while covering the original state-action space as much as possible. As a result, model learning becomes easier and more accurate, which allows robust, stable model-based planning or model-based RL. This transformation is learned in an unsupervised manner, before any task is specified by the user. Downstream tasks can then be solved with model-based control in a zero-shot fashion, without additional environment interactions. We theoretically analyze PMA and empirically demonstrate that PMA leads to significant improvements over prior unsupervised model-based RL approaches in a range of benchmark environments. Our code and videos are available at https://seohong.me/projects/pma/

翻译：基于模型的强化学习（RL）的核心组件是用于预测动作结果的动力学模型。该预测模型的误差会降低基于模型控制器的性能，而复杂马尔可夫决策过程（MDP）会带来极难解决的预测问题。为缓解该问题，我们提出可预测MDP抽象（PMA）：并非在原始MDP上训练预测模型，而是在具有学习动作空间的变换后MDP上训练模型——该空间仅允许可预测、易建模的动作，同时尽可能覆盖原始状态-动作空间。由此，模型学习变得更简单且更精确，从而实现鲁棒、稳定的基于模型的规划或基于模型的强化学习。该变换在用户指定任务前以无监督方式学习完成。下游任务可通过零样本方式基于模型的控制求解，无需额外环境交互。我们从理论上分析了PMA，并通过实验证明PMA在多个基准环境中较先前的无监督基于模型强化学习方法有显著改进。我们的代码和视频见https://seohong.me/projects/pma/