Large Language Models (LLMs) exhibit remarkably powerful capabilities. One of the crucial factors to achieve success is aligning the LLM's output with human preferences. This alignment process often requires only a small amount of data to efficiently enhance the LLM's performance. While effective, research in this area spans multiple domains, and the methods involved are relatively complex to understand. The relationships between different methods have been under-explored, limiting the development of the preference alignment. In light of this, we break down the existing popular alignment strategies into different components and provide a unified framework to study the current alignment strategies, thereby establishing connections among them. In this survey, we decompose all the strategies in preference learning into four components: model, data, feedback, and algorithm. This unified view offers an in-depth understanding of existing alignment algorithms and also opens up possibilities to synergize the strengths of different strategies. Furthermore, we present detailed working examples of prevalent existing algorithms to facilitate a comprehensive understanding for the readers. Finally, based on our unified perspective, we explore the challenges and future research directions for aligning large language models with human preferences.

翻译：大语言模型（LLMs）展现出极其强大的能力。其成功的关键因素之一在于将大语言模型的输出与人类偏好对齐。这一对齐过程通常仅需少量数据即可有效提升大语言模型的性能。尽管方法有效，该领域的研究横跨多个学科，且所涉及的方法相对复杂难懂。不同方法之间的关联尚未得到充分探索，这限制了对齐技术的发展。鉴于此，我们将现有主流对齐策略分解为不同组成部分，并提出一个统一框架来研究当前的对齐策略，从而建立它们之间的联系。在本综述中，我们将偏好学习中的所有策略分解为四个组成部分：模型、数据、反馈和算法。这一统一视角不仅提供了对现有对齐算法的深入理解，也为融合不同策略的优势开辟了可能性。此外，我们详细展示了当前主流算法的工作示例，以帮助读者全面理解。最后，基于我们的统一视角，我们探讨了将大语言模型与人类偏好对齐所面临的挑战及未来研究方向。