Reinforcement learning (RL) using world models has found significant recent successes. However, when a sudden change to world mechanics or properties occurs then agent performance and reliability can dramatically decline. We refer to the sudden change in visual properties or state transitions as novelties. Implementing novelty detection within generated world model frameworks is a crucial task for protecting the agent when deployed. In this paper, we propose straightforward bounding approaches to incorporate novelty detection into world model RL agents, by utilizing the misalignment of the world model's hallucinated states and the true observed states as an anomaly score. We provide effective approaches to detecting novelties in a distribution of transitions learned by an agent in a world model. Finally, we show the advantage of our work in a novel environment compared to traditional machine learning novelty detection methods as well as currently accepted RL focused novelty detection algorithms.

翻译：利用世界模型的强化学习近期取得了显著成功。然而，当世界机制或属性发生突然变化时，智能体的性能和可靠性可能急剧下降。我们将视觉属性或状态转移的突然变化称为新颖性。在生成的世界模型框架内实现新颖性检测，是保障部署中智能体的关键任务。本文提出将新颖性检测融入世界模型强化学习智能体的直接边界方法，通过利用世界模型幻觉状态与真实观测状态之间的偏差作为异常得分。我们提供了有效方法来检测智能体在世界模型中学习到的转移分布中的新颖性。最后，我们展示了在新型环境中，与传统机器学习新颖性检测方法以及当前学界认可的强化学习领域新颖性检测算法相比，本工作的优势。