Long context understanding remains challenging for large language models due to their limited context windows. This paper introduces Long Input Fine-Tuning (LIFT), a novel framework for long-context modeling that can enhance the long-context performance of arbitrary short-context LLMs by dynamically adapting their parameters to the given long input. Importantly, rather than endlessly extending the context window size to accommodate increasingly longer inputs in context, LIFT stores and absorbs the long input in parameters. By fine-tuning the long input into model parameters, LIFT allows short-context LLMs to answer questions even when the required information is not provided in the context during inference, avoiding the quadratic complexity w.r.t. input length of a normal long context model. Furthermore, LIFT does not simply perform continued pretraining on new, long contexts, but leverages carefully designed LLM-generated synthetic tasks to enhance the comprehension of long contexts, moving beyond mere memorization. To accommodate the additional cost of fine-tuning, we design a highly optimized pipeline that reduces the Time to First Token (TTFT) to less than 10 seconds for 8k context. We further provide a comprehensive analysis of LIFT's strengths and limitations in long-context understanding, discuss its feasibility for large-scale real-world deployment, and highlight valuable directions for future research.

翻译：长上下文理解对于大型语言模型而言仍具挑战性，这主要源于其有限的上下文窗口。本文提出长输入微调（LIFT），这是一种新颖的长上下文建模框架，能够通过动态调整模型参数以适应给定的长输入，从而增强任意短上下文LLMs的长上下文性能。重要的是，LIFT并非无止境地扩展上下文窗口大小以容纳越来越长的上下文输入，而是将长输入存储并吸收到模型参数中。通过将长输入微调进模型参数，LIFT使得短上下文LLMs在推理时即使所需信息未在上下文中提供也能回答问题，从而避免了常规长上下文模型随输入长度呈二次方增长的复杂度。此外，LIFT并非简单地在新的长上下文上继续预训练，而是利用精心设计的LLM生成合成任务来增强对长上下文的理解，超越了单纯的记忆。为适应微调带来的额外成本，我们设计了一个高度优化的流水线，将处理8k上下文的首次令牌生成时间（TTFT）缩短至10秒以内。我们进一步全面分析了LIFT在长上下文理解方面的优势与局限，探讨了其大规模实际部署的可行性，并指出了未来研究的有价值方向。