Deep reinforcement learning agents are notoriously sample inefficient, which considerably limits their application to real-world problems. Recently, many model-based methods have been designed to address this issue, with learning in the imagination of a world model being one of the most prominent approaches. However, while virtually unlimited interaction with a simulated environment sounds appealing, the world model has to be accurate over extended periods of time. Motivated by the success of Transformers in sequence modeling tasks, we introduce IRIS, a data-efficient agent that learns in a world model composed of a discrete autoencoder and an autoregressive Transformer. With the equivalent of only two hours of gameplay in the Atari 100k benchmark, IRIS achieves a mean human normalized score of 1.046, and outperforms humans on 10 out of 26 games, setting a new state of the art for methods without lookahead search. To foster future research on Transformers and world models for sample-efficient reinforcement learning, we release our code and models at https://github.com/eloialonso/iris.

翻译：深度强化学习智能体因样本效率低下而闻名，这严重限制了其在现实问题中的应用。近年来，许多基于模型的方法被设计来解决这一问题，其中在想象的世界模型中学习是最主要的方法之一。然而，尽管与模拟环境进行几乎无限的交互听起来颇具吸引力，但世界模型必须在长时间内保持准确性。受Transformer在序列建模任务中成功的启发，我们提出了IRIS，这是一种数据高效的智能体，它在由离散自编码器和自回归Transformer构成的世界模型中学习。在Atari 100k基准测试中，仅需相当于两小时的 gameplay 时间，IRIS便达到了1.046的平均人类标准化得分，并在26款游戏中于10款上超越人类，为无需前瞻搜索的方法设立了新的最佳性能。为促进未来关于Transformer和世界模型在样本高效强化学习中的研究，我们已在https://github.com/eloialonso/iris 上发布代码和模型。