Large language models (LLMs) have achieved remarkable success but still tend to generate factually erroneous responses, a phenomenon known as hallucination. A recent trend is to use preference learning to fine-tune models to align with factuality. However, existing work primarily evaluates fine-tuned models on in-domain (ID) datasets and the factuality on out-of-domain (OOD) datasets remains underexplored. In this paper, we conduct a comprehensive evaluation of the factuality of different models tuned by various preference learning algorithms and demonstrate that their performance on OOD datasets either increases minimally or decreases. Subsequently, we reveal that the main cause of model's failure to uphold factuality under a distribution shift is \textbf{under-alignment}, rather than \textbf{over-alignment}, by analyzing the token distribution shift of the models before and after tuning. Finally, we propose \textbf{APEFT} (\textbf{A}tomic \textbf{P}reference \textbf{E}nhanced \textbf{F}actuality \textbf{T}uning), a framework that enhances model's awareness of factuality at the granularity of individual facts. Extensive experiments demonstrate that APEFT improves model performance by an average of $\boldsymbol{3.45\%}$ on both ID and OOD datasets, which is highly effective.

翻译：大型语言模型（LLMs）已取得显著成功，但仍倾向于生成事实错误的响应，这种现象被称为幻觉。近期趋势是利用偏好学习对模型进行微调，以使其与事实性对齐。然而，现有工作主要评估微调模型在领域内（ID）数据集上的表现，其在领域外（OOD）数据集上的事实性仍未得到充分探索。本文中，我们对通过不同偏好学习算法调优的各类模型的事实性进行了全面评估，并证明它们在OOD数据集上的性能要么仅小幅提升，要么出现下降。随后，通过分析模型调优前后的词元分布偏移，我们揭示了模型在分布偏移下未能保持事实性的主要原因是**欠对齐**，而非**过对齐**。最后，我们提出了**APEFT**（**A**tomic **P**reference **E**nhanced **F**actuality **T**uning），一个在单个事实粒度上增强模型事实性感知能力的框架。大量实验表明，APEFT在ID和OOD数据集上平均提升了$\boldsymbol{3.45\%}$的模型性能，效果显著。