Large language models (LLMs), despite their powerful capabilities, suffer from factual hallucinations where they generate verifiable falsehoods. We identify a root of this issue: the imbalanced data distribution in the pretraining corpus, which leads to a state of "low-probability truth" and "high-probability falsehood". Recent approaches, such as teaching models to say "I don't know" or post-hoc knowledge editing, either evade the problem or face catastrophic forgetting. To address this issue from its root, we propose \textbf{PretrainRL}, a novel framework that integrates reinforcement learning into the pretraining phase to consolidate factual knowledge. The core principle of PretrainRL is "\textbf{debiasing then learning}." It actively reshapes the model's probability distribution by down-weighting high-probability falsehoods, thereby making "room" for low-probability truths to be learned effectively. To enable this, we design an efficient negative sampling strategy to discover these high-probability falsehoods and introduce novel metrics to evaluate the model's probabilistic state concerning factual knowledge. Extensive experiments on three public benchmarks demonstrate that PretrainRL significantly alleviates factual hallucinations and outperforms state-of-the-art methods.

翻译：大语言模型（LLMs）尽管能力强大，却存在事实性幻觉问题，即生成可验证的错误信息。我们识别出该问题的一个根源：预训练语料库中不平衡的数据分布，导致了“低概率真实”与“高概率虚假”的状态。现有方法，例如教导模型回答“我不知道”或进行事后知识编辑，要么回避了问题，要么面临灾难性遗忘。为了从根本上解决此问题，我们提出了 \textbf{PretrainRL}，一个将强化学习整合到预训练阶段以巩固事实知识的新框架。PretrainRL 的核心原则是“\textbf{去偏后学习}”。它通过降低高概率虚假信息的权重，主动重塑模型的概率分布，从而为有效学习低概率真实信息“腾出空间”。为此，我们设计了一种高效的负采样策略来发现这些高概率虚假信息，并引入了新颖的指标来评估模型关于事实知识的概率状态。在三个公开基准上的大量实验表明，PretrainRL 显著缓解了事实性幻觉，并优于现有最先进方法。