Transformer architectures have facilitated the development of large-scale and general-purpose sequence models for prediction tasks in natural language processing and computer vision, e.g., GPT-3 and Swin Transformer. Although originally designed for prediction problems, it is natural to inquire about their suitability for sequential decision-making and reinforcement learning problems, which are typically beset by long-standing issues involving sample efficiency, credit assignment, and partial observability. In recent years, sequence models, especially the Transformer, have attracted increasing interest in the RL communities, spawning numerous approaches with notable effectiveness and generalizability. This survey presents a comprehensive overview of recent works aimed at solving sequential decision-making tasks with sequence models such as the Transformer, by discussing the connection between sequential decision-making and sequence modeling, and categorizing them based on the way they utilize the Transformer. Moreover, this paper puts forth various potential avenues for future research intending to improve the effectiveness of large sequence models for sequential decision-making, encompassing theoretical foundations, network architectures, algorithms, and efficient training systems. As this article has been accepted by the Frontiers of Computer Science, here is an early version, and the most up-to-date version can be found at https://journal.hep.com.cn/fcs/EN/10.1007/s11704-023-2689-5

翻译：Transformer架构推动了自然语言处理和计算机视觉中预测任务的大规模通用序列模型的发展，例如GPT-3和Swin Transformer。尽管这些模型最初是为预测问题设计的，但自然引发了一个问题：它们是否适用于序贯决策和强化学习问题——这类问题通常面临样本效率、信用分配和部分可观测性等长期存在的挑战。近年来，序列模型（尤其是Transformer）在强化学习社区中引起了日益增长的兴趣，催生了许多具有显著有效性和泛化能力的方法。本综述通过探讨序贯决策与序列建模之间的联系，并根据利用Transformer的不同方式对现有工作进行分类，全面概述了近年来利用Transformer等序列模型解决序贯决策任务的研究。此外，本文提出了若干未来研究方向，旨在提升大规模序列模型在序贯决策中的有效性，涵盖理论基础、网络架构、算法和高效训练系统。本文已被《Frontiers of Computer Science》接收，此版本为早期版本，最新版本可见https://journal.hep.com.cn/fcs/EN/10.1007/s11704-023-2689-5