Aligning large language models (LLMs) with human preferences is crucial for enhancing their utility in terms of helpfulness, truthfulness, safety, harmlessness, and interestingness. Existing methods for achieving this alignment often involves employing reinforcement learning from human feedback (RLHF) to fine-tune LLMs based on human labels assessing the relative quality of model responses. Nevertheless, RLHF is susceptible to instability during fine-tuning and presents challenges in implementation.Drawing inspiration from the emerging field of representation engineering (RepE), this study aims to identify relevant representations for high-level human preferences embedded in patterns of activity within an LLM, and achieve precise control of model behavior by transforming its representations. This novel approach, denoted as Representation Alignment from Human Feedback (RAHF), proves to be effective, computationally efficient, and easy to implement.Extensive experiments demonstrate the efficacy of RAHF in not only capturing but also manipulating representations to align with a broad spectrum of human preferences or values, rather than being confined to a singular concept or function (e.g. honesty or bias). RAHF's versatility in accommodating diverse human preferences shows its potential for advancing LLM performance.

翻译：将大型语言模型（LLMs）与人类偏好对齐，对于提升其在有用性、真实性、安全性、无害性和趣味性方面的效用至关重要。现有的对齐方法通常采用基于人类反馈的强化学习（RLHF），根据评估模型回答相对质量的人类标签来微调LLMs。然而，RLHF在微调过程中容易不稳定，且实施起来颇具挑战。受新兴的表征工程（RepE）领域启发，本研究旨在识别嵌入在LLM活动模式中的高层次人类偏好的相关表征，并通过转换其表征来实现对模型行为的精确控制。这种新颖方法，即基于人类反馈的表征对齐（RAHF），被证明有效、计算高效且易于实现。大量实验表明，RAHF不仅能有效捕获，还能操控表征，使其与广泛的人类偏好或价值观对齐，而非局限于单一概念或功能（如诚实性或偏见）。RAHF在适应多样化人类偏好方面的多功能性，展现其在提升LLM性能方面的潜力。