Large language models have an exceptional capability to incorporate new information in a contextual manner. However, the full potential of such an approach is often restrained due to a limitation in the effective context length. One solution to this issue is to endow an attention layer with access to an external memory, which comprises of (key, value) pairs. Yet, as the number of documents increases, the proportion of relevant keys to irrelevant ones decreases, leading the model to focus more on the irrelevant keys. We identify a significant challenge, dubbed the distraction issue, where keys linked to different semantic values might overlap, making them hard to distinguish. To tackle this problem, we introduce the Focused Transformer (FoT), a technique that employs a training process inspired by contrastive learning. This novel approach enhances the structure of the (key, value) space, enabling an extension of the context length. Our method allows for fine-tuning pre-existing, large-scale models to lengthen their effective context. This is demonstrated by our fine-tuning of $3B$ and $7B$ OpenLLaMA checkpoints. The resulting models, which we name LongLLaMA, exhibit advancements in tasks requiring a long context. We further illustrate that our LongLLaMA models adeptly manage a $256 k$ context length for passkey retrieval.

翻译：大型语言模型具有以上下文方式整合新信息的卓越能力。然而，这种方法的全部潜力往往因有效上下文长度的限制而受限。解决该问题的一种方案是为注意力层赋予访问外部存储器的能力，该存储器包含（键，值）对。但随着文档数量增加，相关键与无关键的比例下降，导致模型更关注无关键。我们发现了一个称为"分心问题"的重大挑战，即关联不同语义值的键可能重叠，使其难以区分。为解决此问题，我们提出了聚焦Transformer（FoT），该技术采用受对比学习启发的训练过程。这种新颖方法增强了（键，值）空间的结构，从而能够扩展上下文长度。我们的方法允许对预训练的大规模模型进行微调以延长其有效上下文。我们通过对$3B$和$7B$ OpenLLaMA检查点的微调证明了这一点。由此得到的模型（命名为LongLLaMA）在需要长上下文的任务中展现了进步。我们进一步证明，我们的LongLLaMA模型能够熟练管理$256k$的上下文长度以进行密码检索。