Modeling long texts has been an essential technique in the field of natural language processing (NLP). With the ever-growing number of long documents, it is important to develop effective modeling methods that can process and analyze such texts. However, long texts pose important research challenges for existing text models, with more complex semantics and special characteristics. In this paper, we provide an overview of the recent advances on long texts modeling based on Transformer models. Firstly, we introduce the formal definition of long text modeling. Then, as the core content, we discuss how to process long input to satisfy the length limitation and design improved Transformer architectures to effectively extend the maximum context length. Following this, we discuss how to adapt Transformer models to capture the special characteristics of long texts. Finally, we describe four typical applications involving long text modeling and conclude this paper with a discussion of future directions. Our survey intends to provide researchers with a synthesis and pointer to related work on long text modeling.

翻译：长文本建模一直是自然语言处理（NLP）领域中的关键技术。随着长文档数量的持续增长，开发能够处理和分析此类文本的有效建模方法至关重要。然而，长文本具有更复杂的语义和特殊特性，为现有文本模型带来了重要的研究挑战。本文系统综述了基于Transformer模型的长文本建模最新进展。首先，我们给出了长文本建模的形式化定义。其次，作为核心内容，我们讨论了如何通过处理长输入以满足长度限制，并设计改进的Transformer架构以有效扩展最大上下文长度。随后，我们探讨了如何适配Transformer模型以捕捉长文本的特殊特性。最后，我们描述了涉及长文本建模的四类典型应用，并在讨论未来研究方向后总结了本文。本综述旨在为研究人员提供长文本建模相关工作的综合概述与指引。