MuZero has achieved superhuman performance in various games by using a dynamics network to predict environment dynamics for planning, without relying on simulators. However, the latent states learned by the dynamics network make its planning process opaque. This paper aims to demystify MuZero's model by interpreting the learned latent states. We incorporate observation reconstruction and state consistency into MuZero training and conduct an in-depth analysis to evaluate latent states across two board games: 9x9 Go and Outer-Open Gomoku, and three Atari games: Breakout, Ms. Pacman, and Pong. Our findings reveal that while the dynamics network becomes less accurate over longer simulations, MuZero still performs effectively by using planning to correct errors. Our experiments also show that the dynamics network learns better latent states in board games than in Atari games. These insights contribute to a better understanding of MuZero and offer directions for future research to improve the playing performance, robustness, and interpretability of the MuZero algorithm.

翻译：MuZero通过使用动态网络预测环境动态进行规划，无需依赖模拟器，已在多种游戏中实现超越人类的表现。然而，动态网络学习到的潜在状态使其规划过程不透明。本文旨在通过解读学习到的潜在状态来揭示MuZero模型的内在机制。我们将观测重建和状态一致性纳入MuZero训练，并深入分析评估了两种棋盘游戏（9x9围棋和Outer-Open五子棋）以及三款Atari游戏（Breakout、Ms. Pacman和Pong）中的潜在状态。研究发现，尽管动态网络在较长模拟中的准确性会下降，但MuZero仍能通过规划纠正误差而保持有效性能。实验还表明，动态网络在棋盘游戏中学习的潜在状态优于Atari游戏。这些发现有助于深入理解MuZero算法，并为未来提升其游戏性能、鲁棒性和可解释性提供了研究方向。