Extending the context window of large language models (LLMs) is getting popular recently, while the solution of augmenting LLMs with retrieval has existed for years. The natural questions are: i) Retrieval-augmentation versus long context window, which one is better for downstream tasks? ii) Can both methods be combined to get the best of both worlds? In this work, we answer these questions by studying both solutions using two state-of-the-art pretrained LLMs, i.e., a proprietary 43B GPT and LLaMA2-70B. Perhaps surprisingly, we find that LLM with 4K context window using simple retrieval-augmentation at generation can achieve comparable performance to finetuned LLM with 16K context window via positional interpolation on long context tasks, while taking much less computation. More importantly, we demonstrate that retrieval can significantly improve the performance of LLMs regardless of their extended context window sizes. Our best model, retrieval-augmented LLaMA2-70B with 32K context window, outperforms GPT-3.5-turbo-16k and Davinci003 in terms of average score on seven long context tasks including question answering and query-based summarization. It also outperforms its non-retrieval LLaMA2-70B-32k baseline by a margin, while being much faster at generation. Our study provides general insights on the choice of retrieval-augmentation versus long context extension of LLM for practitioners.

翻译：近年来，扩展大型语言模型（LLM）的上下文窗口逐渐流行，而通过检索增强LLM的方案已存在多年。自然引发两个问题：i）对于下游任务，检索增强与长上下文窗口哪种方法更优？ii）两者能否结合以实现优势互补？本研究通过使用两种最先进的预训练LLM（即专有43B GPT模型和LLaMA2-70B）对上述方案进行系统比较。令人惊讶的是，我们发现：在生成阶段采用简单检索增强的4K上下文窗口LLM，其性能可媲美通过位置插值微调得到的16K上下文窗口LLM，且计算量显著降低。更重要的是，无论扩展上下文窗口大小如何，检索均能有效提升LLM性能。我们的最优模型——采用32K上下文窗口的检索增强型LLaMA2-70B，在包含问答和基于查询的摘要等七项长上下文任务中，平均得分超越GPT-3.5-turbo-16k和Davinci003。与无检索的LLaMA2-70B-32k基线相比，该模型在生成速度大幅提升的同时仍保持显著优势。本研究为实践者在检索增强与长上下文扩展之间的选择提供了通用性指导。