Alignment is a standard procedure to fine-tune pre-trained large language models (LLMs) to follow natural language instructions and serve as helpful AI assistants. We have observed, however, that the conventional alignment process fails to enhance the factual accuracy of LLMs, and often leads to the generation of more false facts (i.e. hallucination). In this paper, we study how to make the LLM alignment process more factual, by first identifying factors that lead to hallucination in both alignment steps:\ supervised fine-tuning (SFT) and reinforcement learning (RL). In particular, we find that training the LLM on new knowledge or unfamiliar texts can encourage hallucination. This makes SFT less factual as it trains on human labeled data that may be novel to the LLM. Furthermore, reward functions used in standard RL can also encourage hallucination, because it guides the LLM to provide more helpful responses on a diverse set of instructions, often preferring longer and more detailed responses. Based on these observations, we propose factuality-aware alignment, comprised of factuality-aware SFT and factuality-aware RL through direct preference optimization. Experiments show that our proposed factuality-aware alignment guides LLMs to output more factual responses while maintaining instruction-following capability.

翻译：对齐是一种标准流程，用于微调预训练大语言模型，使其遵循自然语言指令并成为有用的AI助手。然而，我们观察到，传统的对齐流程未能提升大语言模型的事实准确性，反而常导致生成更多虚假事实（即幻觉）。本文通过首先识别对齐步骤——监督微调和强化学习中导致幻觉的因素，研究如何使大语言模型对齐过程更具事实性。具体而言，我们发现，在大语言模型接触新知识或不熟悉文本时进行训练会加剧幻觉。这使得监督微调的事实性降低，因其训练所依据的人工标注数据对模型而言可能是全新的。此外，标准强化学习中使用的奖励函数也可能加剧幻觉，因为它引导模型针对多样化的指令生成更有帮助的响应，通常偏好更长和更详细的输出。基于这些观察，我们提出面向事实感知的对齐方法，包括通过直接偏好优化实现的面向事实感知的监督微调与面向事实感知的强化学习。实验表明，我们提出的面向事实感知的对齐方法能够引导大语言模型生成更符合事实的响应，同时保持其指令遵循能力。