Recently, with the emergence of numerous Large Language Models (LLMs), the implementation of AI has entered a new era. Irrespective of these models' own capacity and structure, there is a growing demand for LLMs to possess enhanced comprehension of longer and more complex contexts with relatively smaller sizes. Models often encounter an upper limit when processing sequences of sentences that extend beyond their comprehension capacity and result in off-topic or even chaotic responses. While several recent works attempt to address this issue in various ways, they rarely focus on "why models are unable to compensate or strengthen their capabilities on their own". In this paper, we thoroughly investigate the nature of information transfer within LLMs and propose a novel technique called Attention Transition. This technique empowers models to achieve longer and better context comprehension with minimal additional training or impact on generation fluency. Our experiments are conducted in XSum and achieve substantial improvement compared with the original generation results.

翻译：近期，随着海量大型语言模型（LLM）的涌现，人工智能的应用进入崭新时代。无论模型自身的能力与架构如何，业界对较小规模LLM在理解更长、更复杂上下文方面的需求日益增长。当处理超出其理解能力的连续句子序列时，模型常因能力上限而出现偏离主题甚至输出混乱的响应。尽管近年有研究尝试通过多种方式解决该问题，但鲜有聚焦于"模型为何无法自主弥补或强化自身能力"这一根源。本文深入探究LLM内部信息传递的本质，并提出一种名为"注意力过渡"（Attention Transition）的新技术。该技术使模型在极少额外训练或影响生成流畅度的前提下，实现更长、更优的上下文理解。我们在XSum数据集上开展的实验表明，相较于原始生成结果，本方法取得了显著改进。