As the capabilities of large language models (LLMs) have expanded dramatically, aligning these models with human values presents a significant challenge, posing potential risks during deployment. Traditional alignment strategies rely heavily on human intervention, such as Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), or on the self-alignment capacities of LLMs, which usually require a strong LLM's emergent ability to improve its original bad answer. To address these challenges, we propose a novel self-alignment method that utilizes a Chain of Thought (CoT) approach, termed AlignCoT. This method encompasses stages of Question Analysis, Answer Guidance, and Safe Answer production. It is designed to enable LLMs to generate high-quality, safe responses throughout various stages of their development. Furthermore, we introduce the Mixture of insighTful Experts (MoTE) architecture, which applies the mixture of experts to enhance each component of the AlignCoT process, markedly increasing alignment efficiency. The MoTE approach not only outperforms existing methods in aligning LLMs with human values but also highlights the benefits of using self-generated data, revealing the dual benefits of improved alignment and training efficiency.

翻译：随着大语言模型（LLMs）能力的显著扩展，使这些模型与人类价值观对齐成为一项重大挑战，在部署过程中可能带来潜在风险。传统的对齐策略严重依赖人工干预，例如监督微调（SFT）和基于人类反馈的强化学习（RLHF），或者依赖LLMs的自对齐能力，而这通常需要强大的LLM涌现能力来改进其初始的不良回答。为应对这些挑战，我们提出了一种新颖的自对齐方法，即利用思维链（CoT）技术，称为AlignCoT。该方法涵盖问题分析、答案引导和安全答案生成等阶段，旨在使LLMs在其开发的各个阶段都能生成高质量、安全的响应。此外，我们引入了混合洞察专家（MoTE）架构，该架构应用专家混合技术来增强AlignCoT流程的每个组件，显著提升对齐效率。MoTE方法不仅在使LLMs与人类价值观对齐方面优于现有方法，还凸显了使用自生成数据的优势，揭示了在对齐效果和训练效率方面的双重益处。