When applied for processing long text, Large Language Models (LLMs) are limited by their context window. Existing efforts to address this limitation involve training specialized architectures, and cannot be easily applied to off-the-shelf LLMs. We present Parallel Context Windows (PCW), a method that alleviates the context window restriction for any off-the-shelf LLM without further training. The key to the approach is to carve a long context into chunks (``windows''), restrict the attention mechanism to apply only within each window, and re-use the positional embeddings across the windows. Our main results test the PCW approach on in-context learning with models that range in size between 750 million and 178 billion parameters, and show substantial improvements for tasks with diverse input and output spaces. We show additional benefits in other settings where long context windows may be beneficial: multi-hop questions and retrieval-augmented question answering with multiple retrieved documents. Our results highlight Parallel Context Windows as a promising method for applying off-the-shelf LLMs in a range of settings that require long text sequences. We make our code publicly available at https://github.com/ai21labs/parallel-context-windows.

翻译：当应用于处理长文本时，大语言模型（LLMs）受限于其上下文窗口。现有解决该限制的工作涉及训练专用架构，且难以直接应用于现成的大语言模型。我们提出并行上下文窗口（Parallel Context Windows，PCW），一种无需额外训练即可缓解任何现成大语言模型上下文窗口限制的方法。该方法的关键在于将长文本切分为块（即“窗口”），限制注意力机制仅在每个窗口内应用，并在窗口间复用位置嵌入。我们的主要实验在参数规模介于7.5亿至1780亿之间的模型上进行上下文学习测试，结果表明该方法在输入输出空间多样的任务上取得了显著改进。我们还展示了长上下文窗口可能有益的其他场景中的额外优势：多跳问题以及基于多个检索文档的检索增强问答。我们的结果凸显了并行上下文窗口作为一种有前景的方法，可用于需要长文本序列的多种场景中现成大语言模型的应用。我们已公开代码：https://github.com/ai21labs/parallel-context-windows。