Truthfulness is paramount for large language models (LLMs) as they are increasingly deployed in real-world applications. However, existing LLMs still struggle with generating truthful answers and content, as evidenced by their modest performance on benchmarks like TruthfulQA. To address this issue, we propose GRAdual self-truTHifying (GRATH), a novel post-processing method to enhance truthfulness of LLMs. GRATH utilizes out-of-domain question prompts to generate corresponding answers and adaptively optimizes the model via direct preference optimization (DPO). Note that during this process, GRATH learns truthfulness in a self-supervised manner without requiring annotated answers. In particular, GRATH first generates pairwise truthfulness training data by prompting the LLM itself, with each pair containing a question and its correct and incorrect answers. The model is then fine-tuned using DPO to learn from the difference between answer pairs. Subsequently, GRATH iteratively refines the truthfulness data and optimizes the model, leading to a gradual improvement in model truthfulness. Empirically, we evaluate GRATH using different 7B-LLMs and compare with LLMs with similar or even larger sizes on benchmark datasets. Our results show that GRATH effectively improves LLMs' truthfulness without compromising other core capabilities. Notably, GRATH achieves state-of-the-art performance on TruthfulQA, with MC1 accuracy as 54.71% and MC2 accuracy as 69.10%, which even surpass those on larger-scale models, such as Llama2-Chat-70B, by 23.62% and 24.18%, respectively.

翻译：真实性问题对于大语言模型（LLMs）至关重要，因为此类模型正越来越多地应用于实际场景中。然而，现有LLMs在生成真实答案和内容方面仍存在困难，这一点在TruthfulQA等基准测试中表现出的中等性能上得到印证。为解决这一问题，我们提出一种新型后处理方法——渐进式自我真实化（GRATH），旨在提升LLMs的真实性。GRATH利用领域外的问题提示生成对应答案，并通过直接偏好优化（DPO）自适应地优化模型。值得注意的是，在此过程中，GRATH以自监督方式学习真实性，无需依赖标注答案。具体而言，GRATH首先通过提示LLM自身生成成对真实性训练数据，每对数据包含一个问题及其正确与错误答案；随后使用DPO对模型进行微调，使其从答案对的差异中学习。接着，GRATH迭代地优化真实性数据并改进模型，从而实现模型真实性的渐进式提升。在实验层面，我们使用不同7B规模的LLMs评估GRATH，并与基准数据集上规模相似或更大的LLMs进行比较。结果表明，GRATH在提升LLMs真实性的同时，未损害其他核心能力。值得注意的是，GRATH在TruthfulQA上达到了最优性能，MC1准确率为54.71%，MC2准确率为69.10%，甚至分别超越大规模模型（如Llama2-Chat-70B）23.62%和24.18%。