Transformers have significantly impacted domains like natural language processing, computer vision, and robotics, where they improve performance compared to other neural networks. This survey explores how transformers are used in reinforcement learning (RL), where they are seen as a promising solution for addressing challenges such as unstable training, credit assignment, lack of interpretability, and partial observability. We begin by providing a brief domain overview of RL, followed by a discussion on the challenges of classical RL algorithms. Next, we delve into the properties of the transformer and its variants and discuss the characteristics that make them well-suited to address the challenges inherent in RL. We examine the application of transformers to various aspects of RL, including representation learning, transition and reward function modeling, and policy optimization. We also discuss recent research that aims to enhance the interpretability and efficiency of transformers in RL, using visualization techniques and efficient training strategies. Often, the transformer architecture must be tailored to the specific needs of a given application. We present a broad overview of how transformers have been adapted for several applications, including robotics, medicine, language modeling, cloud computing, and combinatorial optimization. We conclude by discussing the limitations of using transformers in RL and assess their potential for catalyzing future breakthroughs in this field.

翻译：Transformer已对自然语言处理、计算机视觉和机器人等领域产生了重大影响，与其他神经网络相比，它们在提升性能方面表现显著。本综述探讨了Transformer在强化学习中的应用，被视为解决训练不稳定、信用分配、缺乏可解释性和部分可观测性等挑战的有前景方案。首先简要概述强化学习领域，并讨论经典强化学习算法面临的挑战；随后深入剖析Transformer及其变体的特性，阐述其适合应对强化学习固有特征的技术优势。我们考察Transformer在强化学习各环节的应用，包括表征学习、转移函数与奖励函数建模及策略优化。同时评述近期研究如何通过可视化技术和高效训练策略提升Transformer在强化学习中的可解释性与效率。由于Transformer架构常需针对具体应用定制，我们全面综述了其在机器人、医学、语言建模、云计算及组合优化等领域的适配应用。最后讨论Transformer在强化学习中应用的局限性，并评估其在催化该领域未来突破方面的潜力。