Deep neural networks have been successful in many reinforcement learning settings. However, compared to human learners they are overly data hungry. To build a sample-efficient world model, we apply a transformer to real-world episodes in an autoregressive manner: not only the compact latent states and the taken actions but also the experienced or predicted rewards are fed into the transformer, so that it can attend flexibly to all three modalities at different time steps. The transformer allows our world model to access previous states directly, instead of viewing them through a compressed recurrent state. By utilizing the Transformer-XL architecture, it is able to learn long-term dependencies while staying computationally efficient. Our transformer-based world model (TWM) generates meaningful, new experience, which is used to train a policy that outperforms previous model-free and model-based reinforcement learning algorithms on the Atari 100k benchmark.

翻译：深度神经网络已在许多强化学习场景中取得成功。然而，与人类学习者相比，它们对数据的需求过高。为了构建样本高效的世界模型，我们将Transformer以自回归方式应用于真实世界的回合数据：不仅包括压缩的潜在状态和已执行的动作，还包括经历或预测的奖励，这些都被输入到Transformer中，使其能够灵活地关注不同时间步长的所有三种模态。Transformer使我们的世界模型能够直接访问先前的状态，而不是通过压缩的循环状态来查看它们。通过利用Transformer-XL架构，该模型能够在保持计算效率的同时学习长期依赖关系。我们基于Transformer的世界模型（TWM）能够生成有意义的新经验，这些经验用于训练策略，该策略在Atari 100k基准测试中的表现优于之前的无模型和基于模型的强化学习算法。