With the bomb ignited by ChatGPT, Transformer-based Large Language Models (LLMs) have paved a revolutionary path toward Artificial General Intelligence (AGI) and have been applied in diverse areas as knowledge bases, human interfaces, and dynamic agents. However, a prevailing limitation exists: many current LLMs, constrained by resources, are primarily pre-trained on shorter texts, rendering them less effective for longer-context prompts, commonly encountered in real-world settings. In this paper, we present a comprehensive survey focusing on the advancement of model architecture in Transformer-based LLMs to optimize long-context capabilities across all stages from pre-training to inference. We firstly delineate and analyze the problems of handling long-context input and output with the current Transformer-based models. Then, we mainly offer a holistic taxonomy to navigate the landscape of Transformer upgrades on architecture to solve these problems. Afterward, we provide the investigation on wildly used evaluation necessities tailored for long-context LLMs, including datasets, metrics, and baseline models, as well as some amazing optimization toolkits like libraries, systems, and compilers to augment LLMs' efficiency and efficacy across different stages. Finally, we further discuss the predominant challenges and potential avenues for future research in this domain. Additionally, we have established a repository where we curate relevant literature with real-time updates at https://github.com/Strivin0311/long-llms-learning.

翻译：ChatGPT点燃的燎原之火使基于Transformer的大型语言模型（LLMs）开辟了通往通用人工智能（AGI）的革命性道路，并已广泛应用于知识库、人机交互接口及动态智能体等多元领域。然而，当前普遍存在一个局限：受限于计算资源，许多LLMs主要在短文本上进行预训练，导致其在现实场景中常见的超长上下文提示下表现欠佳。本文聚焦Transformer架构演进，系统综述了从预训练到推理全阶段中优化长上下文能力的模型架构创新。我们首先阐析现有Transformer模型在处理长上下文输入输出时面临的困境；继而构建了面向架构升级的全局分类体系，以梳理解决上述问题的技术路线；随后深入调研了专为长上下文LLMs设计的评估基础设施（包含数据集、评估指标与基线模型），以及贯穿各阶段提升模型效率与效能的优化工具包（如函数库、系统与编译器）。最后，我们进一步探讨该领域当前的核心挑战与未来研究方向。此外，我们已建立实时更新的文献资源库（https://github.com/Strivin0311/long-llms-learning）。