Large language models have an exceptional capability to incorporate new information in a contextual manner. However, the full potential of such an approach is often restrained due to a limitation in the effective context length. One solution to this issue is to endow an attention layer with access to an external memory, which comprises of (key, value) pairs. Yet, as the number of documents increases, the proportion of relevant keys to irrelevant ones decreases, leading the model to focus more on the irrelevant keys. We identify a significant challenge, dubbed the distraction issue, where keys linked to different semantic values might overlap, making them hard to distinguish. To tackle this problem, we introduce the Focused Transformer (FoT), a technique that employs a training process inspired by contrastive learning. This novel approach enhances the structure of the (key, value) space, enabling an extension of the context length. Our method allows for fine-tuning pre-existing, large-scale models to lengthen their effective context. This is demonstrated by our fine-tuning of $3B$ and $7B$ OpenLLaMA checkpoints. The resulting models, which we name LongLLaMA, exhibit advancements in tasks requiring a long context. We further illustrate that our LongLLaMA models adeptly manage a $256 k$ context length for passkey retrieval.

翻译：大型语言模型具有在上下文中整合新信息的卓越能力。然而，这种方法的全部潜力往往因有效上下文长度的限制而受到制约。解决该问题的一个途径是赋予注意力层访问外部记忆的能力，该记忆由（键，值）对组成。但随着文档数量增加，相关键与无关键的比例下降，导致模型更倾向于关注无关键。我们识别出一个关键挑战——分散注意问题：当与不同语义值关联的键可能重叠时，区分它们变得困难。为解决该问题，我们提出聚焦Transformer（FoT）技术，其采用受对比学习启发的训练过程。这种新方法增强了（键，值）空间的结构，从而能够扩展上下文长度。我们的方法允许对现有大规模模型进行微调以延长其有效上下文。通过在$3B$和$7B$参数的OpenLLaMA检查点上进行微调，我们展示了该方法的有效性。得到的模型被命名为LongLLaMA，在需要长上下文的任务中展现出性能提升。我们进一步证明，LongLLaMA模型能够优雅地处理$256k$上下文长度的密钥检索任务。