One of the biggest challenges to modern deep reinforcement learning (DRL) algorithms is sample efficiency. Many approaches learn a world model in order to train an agent entirely in imagination, eliminating the need for direct environment interaction during training. However, these methods often suffer from either a lack of imagination accuracy, exploration capabilities, or runtime efficiency. We propose Hieros, a hierarchical policy that learns time abstracted world representations and imagines trajectories at multiple time scales in latent space. Hieros uses an S5 layer-based world model, which predicts next world states in parallel during training and iteratively during environment interaction. Due to the special properties of S5 layers, our method can train in parallel and predict next world states iteratively during imagination. This allows for more efficient training than RNN-based world models and more efficient imagination than Transformer-based world models. We show that our approach outperforms the state of the art in terms of mean and median normalized human score on the Atari 100k benchmark, and that our proposed world model is able to predict complex dynamics very accurately. We also show that Hieros displays superior exploration capabilities compared to existing approaches.

翻译：摘要：现代深度强化学习算法面临的最大挑战之一是样本效率。许多方法通过学习世界模型，完全在想象中训练智能体，从而消除训练期间与环境直接交互的需求。然而，这些方法往往在想象精度、探索能力或运行时效率方面存在不足。我们提出Hieros——一种分层策略，能够学习时间抽象的世界表征，并在潜在空间中跨多个时间尺度想象轨迹。Hieros采用基于S5层的世界模型，该模型在训练期间并行预测下一世界状态，并在环境交互期间迭代预测。由于S5层的特殊性质，我们的方法可在训练时并行处理，并在想象中迭代预测下一世界状态。这使得其训练效率高于基于RNN的世界模型，且想象效率优于基于Transformer的世界模型。实验表明，在Atari 100k基准测试中，我们的方法在归一化人类得分的均值和中位数上均超越了现有最优方法，且所提出的世界模型能够非常精确地预测复杂动态。我们还证明，与现有方法相比，Hieros展现出更优越的探索能力。