Retrieval-Augmented Language Modeling (RALM) methods, which condition a language model (LM) on relevant documents from a grounding corpus during generation, were shown to significantly improve language modeling performance. In addition, they can mitigate the problem of factually inaccurate text generation and provide natural source attribution mechanism. Existing RALM approaches focus on modifying the LM architecture in order to facilitate the incorporation of external information, significantly complicating deployment. This paper considers a simple alternative, which we dub In-Context RALM: leaving the LM architecture unchanged and prepending grounding documents to the input, without any further training of the LM. We show that In-Context RALM that builds on off-the-shelf general purpose retrievers provides surprisingly large LM gains across model sizes and diverse corpora. We also demonstrate that the document retrieval and ranking mechanism can be specialized to the RALM setting to further boost performance. We conclude that In-Context RALM has considerable potential to increase the prevalence of LM grounding, particularly in settings where a pretrained LM must be used without modification or even via API access.

翻译：检索增强语言建模（RALM）方法通过在生成过程中将语言模型（LM）与来自基础语料库的相关文档条件化，已被证明能显著提升语言建模性能。此外，该方法还能缓解事实性文本生成不准确的问题，并提供自然的来源归属机制。现有RALM方法侧重于修改LM架构以促进外部信息的整合，这大大增加了部署复杂性。本文考虑一种简单替代方案，我们称之为上下文RALM：保持LM架构不变，将基础文档预置于输入之前，无需对LM进行任何额外训练。研究表明，基于现成通用检索器构建的上下文RALM，在不同模型规模和多样语料库上均能带来出乎意料的显著LM性能提升。我们还证明，可将文档检索与排序机制专门适配于RALM场景以进一步提升性能。我们得出结论：上下文RALM在增强LM基础性方面具有巨大潜力，尤其在必须使用预训练LM且无法修改其架构、甚至仅能通过API访问的场景下。