Parametric language models (LMs), which are trained on vast amounts of web data, exhibit remarkable flexibility and capability. However, they still face practical challenges such as hallucinations, difficulty in adapting to new data distributions, and a lack of verifiability. In this position paper, we advocate for retrieval-augmented LMs to replace parametric LMs as the next generation of LMs. By incorporating large-scale datastores during inference, retrieval-augmented LMs can be more reliable, adaptable, and attributable. Despite their potential, retrieval-augmented LMs have yet to be widely adopted due to several obstacles: specifically, current retrieval-augmented LMs struggle to leverage helpful text beyond knowledge-intensive tasks such as question answering, have limited interaction between retrieval and LM components, and lack the infrastructure for scaling. To address these, we propose a roadmap for developing general-purpose retrieval-augmented LMs. This involves a reconsideration of datastores and retrievers, the exploration of pipelines with improved retriever-LM interaction, and significant investment in infrastructure for efficient training and inference.

翻译：参数化语言模型（LMs）在海量网络数据上训练，展现出显著的灵活性和能力。然而，它们仍面临实际挑战，如幻觉现象、难以适应新的数据分布以及缺乏可验证性。在本立场论文中，我们主张用检索增强语言模型取代参数化语言模型，作为下一代语言模型。通过在推理过程中整合大规模数据存储，检索增强语言模型能够更可靠、更可适应且更可归因。尽管潜力巨大，但检索增强语言模型因若干障碍尚未被广泛采用：具体而言，当前的检索增强语言模型难以在知识密集型任务（如问答）之外利用有益的文本，检索与语言模型组件之间的交互有限，且缺乏可扩展的基础设施。为解决这些问题，我们提出了一条开发通用检索增强语言模型的路线图。这包括重新审视数据存储和检索器，探索具有改进的检索器-语言模型交互的流水线，并大力投资于高效训练和推理的基础设施。