Transformer-based Large Language Models (LLMs) often impose limitations on the length of the text input to ensure the generation of fluent and relevant responses. This constraint restricts their applicability in scenarios involving long texts. We propose a novel semantic compression method that enables generalization to texts that are 6-8 times longer, without incurring significant computational costs or requiring fine-tuning. Our proposed framework draws inspiration from source coding in information theory and employs a pre-trained model to reduce the semantic redundancy of long inputs before passing them to the LLMs for downstream tasks. Experimental results demonstrate that our method effectively extends the context window of LLMs across a range of tasks including question answering, summarization, few-shot learning, and information retrieval. Furthermore, the proposed semantic compression method exhibits consistent fluency in text generation while reducing the associated computational overhead.

翻译：基于Transformer架构的大语言模型通常对输入文本长度施加限制，以确保生成流畅且相关的回复。这一约束限制了其在长文本场景中的应用。我们提出了一种新颖的语义压缩方法，使模型能够泛化处理长度6-8倍的文本，且无需显著增加计算成本或进行微调。该框架受信息论中信源编码启发，利用预训练模型在将长输入传递给大语言模型执行下游任务前，降低其语义冗余。实验结果表明，我们的方法在问答、摘要、少样本学习和信息检索等多项任务中有效扩展了大语言模型的上下文窗口。此外，所提出的语义压缩方法在保持文本生成流畅性的同时，降低了相关计算开销。