The quality of finetuning data is crucial for aligning large language models (LLMs) with human values. Current methods to improve data quality are either labor-intensive or prone to factual errors caused by LLM hallucinations. This paper explores elevating the quality of existing instruction data to better align with human values, introducing a simple and effective approach named ReAlign, which reformats the responses of instruction data into a format that better aligns with pre-established criteria and the collated evidence. This approach minimizes human annotation, hallucination, and the difficulty in scaling, remaining orthogonal to existing alignment techniques. Experimentally, ReAlign significantly boosts the general alignment ability, math reasoning, factuality, and readability of the LLMs. Encouragingly, without introducing any additional data or advanced training techniques, and merely by reformatting the response, LLaMA-2-13B's mathematical reasoning ability on GSM8K can be improved from 46.77% to 56.63% in accuracy. Additionally, a mere 5% of ReAlign data yields a 67% boost in general alignment ability measured by the Alpaca dataset. This work highlights the need for further research into the science and mechanistic interpretability of LLMs. We have made the associated code and data publicly accessible to support future studies at https://github.com/GAIR-NLP/ReAlign.

翻译：微调数据的质量对于使大型语言模型（LLMs）与人类价值观对齐至关重要。当前提高数据质量的方法要么劳动密集，要么容易因LLM幻觉导致的事实错误。本文探索如何提升现有指令数据的质量以更好地对齐人类价值观，提出了一种简单有效的方法——ReAlign，该方法将指令数据的响应重排为更符合预设标准和整理证据的格式。该方法最小化人工标注、幻觉和扩展难度，与现有对齐技术正交。实验表明，ReAlign显著增强了LLMs的通用对齐能力、数学推理、事实性和可读性。令人鼓舞的是，在不引入任何额外数据或先进训练技术的情况下，仅通过对响应进行重排，LLaMA-2-13B在GSM8K上的数学推理能力准确率即可从46.77%提升至56.63%。此外，仅使用5%的ReAlign数据即可使基于Alpaca数据集测量的通用对齐能力提升67%。本工作强调了进一步研究LLM科学性和机制可解释性的必要性。我们已公开相关代码和数据以支持未来研究，地址为https://github.com/GAIR-NLP/ReAlign。