Large language models often hallucinate with high confidence on "random facts" that lack inferable patterns. We formalize the memorization of such facts as a membership testing problem, unifying the discrete error metrics of Bloom filters with the continuous log-loss of LLMs. By analyzing this problem in the regime where facts are sparse in the universe of plausible claims, we establish a rate-distortion theorem: the optimal memory efficiency is characterized by the minimum KL divergence between score distributions on facts and non-facts. This theoretical framework provides a distinctive explanation for hallucination under an idealized setting: even with optimal training, perfect data, and a simplified ``closed world'' setting, the information-theoretically optimal strategy under limited capacity is not to abstain or forget, but to assign high confidence to some non-facts, resulting in hallucination. We validate this theory empirically on both synthetic and real-world data, showing that hallucinations persist as a natural consequence of lossy compression. The same theorem recovers and sharpens classical space lower bounds for Bloom-type filters, pinning down an additive constant left open for two-sided filters.

翻译：大型语言模型常常对缺乏可推断模式的“随机事实”表现出高置信度的幻觉。我们将此类事实的记忆形式化为成员关系测试问题，统一了布隆过滤器的离散误差度量与大型语言模型的连续对数损失。通过分析事实在可能断言集合中稀疏分布的情景，我们建立了一个率失真定理：最优记忆效率由事实与非事实得分分布之间的最小KL散度刻画。这一理论框架为理想化场景下的幻觉提供了独特解释：即使在最优训练、完美数据以及简化的“封闭世界”设定下，有限容量下信息论最优策略并非弃权或遗忘，而是对某些非事实赋予高置信度，从而导致幻觉。我们在合成数据与真实数据上对理论进行了实证验证，表明幻觉是损压缩的自然结果。该定理同时恢复并强化了布隆型过滤器的经典空间下界，精确确定了双边过滤器中遗留的开常数。