Model-based reinforcement learning has drawn considerable interest in recent years, given its promise to improve sample efficiency. Moreover, when using deep-learned models, it is potentially possible to learn compact models from complex sensor data. However, the effectiveness of these learned models, particularly their capacity to plan, i.e., to improve the current policy, remains unclear. In this work, we study MuZero, a well-known deep model-based reinforcement learning algorithm, and explore how far it achieves its learning objective of a value-equivalent model and how useful the learned models are for policy improvement. Amongst various other insights, we conclude that the model learned by MuZero cannot effectively generalize to evaluate unseen policies, which limits the extent to which we can additionally improve the current policy by planning with the model.

翻译：基于模型的强化学习近年来因有望提高样本效率而备受关注。此外，当使用深度学习的模型时，有可能从复杂的传感器数据中学习紧凑的模型。然而，这些学习模型的有效性，特别是它们进行规划（即改进当前策略）的能力，仍不明确。在这项工作中，我们研究了著名的深度模型强化学习算法MuZero，探讨了它在多大程度上实现了值等价模型的学习目标，以及学习到的模型对策略改进的有用程度。在各种其他发现中，我们得出结论：MuZero学习的模型无法有效泛化以评估未见过的策略，这限制了通过使用模型进行规划来进一步改进当前策略的程度。