Despite their powerful chat, coding, and reasoning abilities, Large Language Models (LLMs) frequently hallucinate. Conventional wisdom suggests that hallucinations are a consequence of a balance between creativity and factuality, which can be mitigated, but not eliminated, by grounding the LLM in external knowledge sources. Through extensive systematic experiments, we show that these traditional approaches fail to explain why LLMs hallucinate in practice. Specifically, we show that LLMs augmented with a massive Mixture of Memory Experts (MoME) can easily memorize large datasets of random numbers. We corroborate these experimental findings with a theoretical construction showing that simple neural networks trained to predict the next token hallucinate when the training loss is above a threshold as it usually does in practice when training on internet scale data. We interpret our findings by comparing against traditional retrieval methods for mitigating hallucinations. We use our findings to design a first generation model for removing hallucinations -- Lamini-1 -- that stores facts in a massive mixture of millions of memory experts that are retrieved dynamically.

翻译：尽管大型语言模型（LLM）在对话、编程和推理方面展现出强大能力，但其频繁产生幻觉的问题依然存在。传统观点认为，幻觉是创造力与事实性之间平衡的结果，通过将LLM与外部知识源进行锚定可以缓解但无法根除该现象。我们通过大量系统性实验表明，这些传统方法无法解释LLM在实际应用中出现幻觉的根本原因。具体而言，我们证明了采用大规模混合记忆专家（MoME）增强的LLM能够轻松记忆海量随机数数据集。我们通过理论构建进一步验证了实验发现：当训练损失超过特定阈值时（这在互联网规模数据训练中普遍存在），即使是仅预测下一标记的简单神经网络也会产生幻觉。我们将这些发现与缓解幻觉的传统检索方法进行对比分析，并基于此设计了首代消除幻觉的模型——Lamini-1。该模型通过动态检索机制，将事实存储于由数百万记忆专家构成的混合记忆系统中。