Big models have achieved revolutionary breakthroughs in the field of AI, but they might also pose potential concerns. Addressing such concerns, alignment technologies were introduced to make these models conform to human preferences and values. Despite considerable advancements in the past year, various challenges lie in establishing the optimal alignment strategy, such as data cost and scalable oversight, and how to align remains an open question. In this survey paper, we comprehensively investigate value alignment approaches. We first unpack the historical context of alignment tracing back to the 1920s (where it comes from), then delve into the mathematical essence of alignment (what it is), shedding light on the inherent challenges. Following this foundation, we provide a detailed examination of existing alignment methods, which fall into three categories: Reinforcement Learning, Supervised Fine-Tuning, and In-context Learning, and demonstrate their intrinsic connections, strengths, and limitations, helping readers better understand this research area. In addition, two emerging topics, personal alignment, and multimodal alignment, are also discussed as novel frontiers in this field. Looking forward, we discuss potential alignment paradigms and how they could handle remaining challenges, prospecting where future alignment will go.

翻译：大模型已在人工智能领域取得革命性突破，但也可能带来潜在风险。为应对这些风险，对齐技术应运而生，旨在使模型符合人类偏好与价值观。尽管过去一年取得了显著进展，但构建最优对齐策略仍面临诸多挑战，如数据成本与可扩展监督问题，如何实现对齐尚未有定论。本文系统综述了价值对齐方法：首先追溯对齐的历史渊源（可溯至20世纪20年代），进而阐明对齐的数学本质及其固有挑战；在此基础上，详细梳理现有三类对齐方法——强化学习、监督微调与上下文学习，并揭示其内在联系、优势与局限性，以助读者深入理解该领域。此外，本文还探讨了个人化对齐与多模态对齐两大新兴前沿方向。展望未来，我们讨论了潜在的对齐范式及其应对剩余挑战的路径，为对齐技术的未来发展方向提供前瞻性思考。