Despite their impressive performance on diverse tasks, large language models (LMs) still struggle with tasks requiring rich world knowledge, implying the limitations of relying solely on their parameters to encode a wealth of world knowledge. This paper aims to understand LMs' strengths and limitations in memorizing factual knowledge, by conducting large-scale knowledge probing experiments of 10 models and 4 augmentation methods on PopQA, our new open-domain QA dataset with 14k questions. We find that LMs struggle with less popular factual knowledge, and that scaling fails to appreciably improve memorization of factual knowledge in the long tail. We then show that retrieval-augmented LMs largely outperform orders of magnitude larger LMs, while unassisted LMs remain competitive in questions about high-popularity entities. Based on those findings, we devise a simple, yet effective, method for powerful and efficient retrieval-augmented LMs, which retrieves non-parametric memories only when necessary. Experimental results show that this significantly improves models' performance while reducing the inference costs.

翻译：尽管大语言模型（LMs）在多种任务上表现惊人，但它们仍难以应对需要丰富世界知识的任务，这暗示了仅依赖其参数编码大量世界知识的局限性。本文旨在通过在大规模知识探测实验中对10个模型和4种增强方法在PopQA（我们新构建的包含14k个问题的开放域问答数据集）上进行评估，来理解语言模型在记忆事实知识方面的优势与局限。我们发现，语言模型在应对流行度较低的事实知识时表现吃力，而模型规模扩大并未显著改善对长尾事实知识的记忆。随后我们证明，检索增强型语言模型在很大程度上优于规模大数个数量级的语言模型，而未经过检索辅助的语言模型在涉及高流行度实体的问题上仍具有竞争力。基于这些发现，我们设计了一种简单但有效的方法来构建强大且高效的检索增强型语言模型，该方法仅在必要时检索非参数化记忆。实验结果表明，该方法在显著提升模型性能的同时降低了推理成本。