Transformer has been considered the dominating neural architecture in NLP and CV, mostly under supervised settings. Recently, a similar surge of using Transformers has appeared in the domain of reinforcement learning (RL), but it is faced with unique design choices and challenges brought by the nature of RL. However, the evolution of Transformers in RL has not yet been well unraveled. In this paper, we seek to systematically review motivations and progress on using Transformers in RL, provide a taxonomy on existing works, discuss each sub-field, and summarize future prospects.

翻译：Transformer已被视为自然语言处理和计算机视觉领域占主导地位的神经架构，主要应用于监督学习场景。近期，类似的使用Transformer的热潮在强化学习领域出现，但受强化学习本质的影响，它面临着独特的设计选择和挑战。然而，Transformer在强化学习中的演进尚未得到充分揭示。本文旨在系统性地综述在强化学习中使用Transformer的动机与进展，对现有工作进行分类，讨论各个子领域，并总结未来展望。