Recently, with the emergence of numerous Large Language Models (LLMs), the implementation of AI has entered a new era. Irrespective of these models' own capacity and structure, there is a growing demand for LLMs to possess enhanced comprehension of longer and more complex contexts with relatively smaller sizes. Models often encounter an upper limit when processing sequences of sentences that extend beyond their comprehension capacity and result in off-topic or even chaotic responses. While several recent works attempt to address this issue in various ways, they rarely focus on "why models are unable to compensate or strengthen their capabilities on their own". In this paper, we thoroughly investigate the nature of information transfer within LLMs and propose a novel technique called Attention Transition. This technique empowers models to achieve longer and better context comprehension with minimal additional training or impact on generation fluency. Our experiments are conducted on the challenging XSum dataset using LLaMa-7b model with context token length ranging from 800 to 1900. Results demonstrate that we achieve substantial improvements compared with the original generation results evaluated by GPT4.

翻译：近期，随着众多大型语言模型（LLMs）的涌现，人工智能的实施已进入新时代。无论这些模型自身的能力与结构如何，业界对LLMs日益增长的需求是：在保持相对较小规模的同时，增强对更长且更复杂上下文的理解能力。当处理超出其理解容量的长句序列时，模型往往会遭遇上限限制，导致生成偏离主题甚至混乱的响应。尽管近期一些工作尝试以不同方式解决该问题，但它们极少关注"模型为何无法自行弥补或强化其能力"这一本质。本文深入探究了LLMs内部信息传递的本质，并提出了一种名为"注意力过渡"（Attention Transition）的新技术。该技术以极少的额外训练代价或不影响生成流畅性为代价，提升了模型对更长、更优上下文的理解能力。我们在具有挑战性的XSum数据集上，使用LLaMa-7b模型（上下文令牌长度范围为800至1900）进行了实验。结果表明，与GPT4评估的原始生成结果相比，我们取得了显著改进。