Aligning large language models (LLMs) with human values is imperative to mitigate potential adverse effects resulting from their misuse. Drawing from the sociological insight that acknowledging all parties' concerns is a key factor in shaping human values, this paper proposes a novel direction to align LLMs by themselves: social scene simulation. To achieve this, we present MATRIX, a novel social scene simulator that emulates realistic scenes around a user's input query, enabling the LLM to take social consequences into account before responding. MATRIX serves as a virtual rehearsal space, akin to a Monopolylogue, where the LLM performs diverse roles related to the query and practice by itself. To inject this alignment, we fine-tune the LLM with MATRIX-simulated data, ensuring adherence to human values without compromising inference speed. We theoretically show that the LLM with MATRIX outperforms Constitutional AI under mild assumptions. Finally, extensive experiments validate that our method outperforms over 10 baselines across 4 benchmarks. As evidenced by 875 user ratings, our tuned 13B-size LLM exceeds GPT-4 in aligning with human values. Code is available at https://github.com/pangxianghe/MATRIX.

翻译：将大型语言模型（LLMs）与人类价值观对齐对于减轻其滥用可能带来的潜在负面影响至关重要。基于社会学见解——承认所有相关方的关切是塑造人类价值观的关键因素——本文提出了一种新颖的自我对齐方向：社会场景模拟。为此，我们提出了MATRIX，一种新颖的社会场景模拟器，它能够模拟用户输入查询周围的真实场景，使LLM在响应前考虑到社会后果。MATRIX作为一个虚拟排练空间，类似于独白剧（Monopolylogue），其中LLM扮演与查询相关的多种角色并自行练习。为了注入这种对齐，我们使用MATRIX模拟的数据对LLM进行微调，确保其遵守人类价值观，同时不牺牲推理速度。我们理论上证明，在温和假设下，采用MATRIX的LLM优于宪法AI（Constitutional AI）。最后，大量实验验证了我们的方法在4个基准测试中优于10多个基线方法。根据875名用户评分，我们微调的13B-size LLM在人类价值观对齐方面超过了GPT-4。代码可在https://github.com/pangxianghe/MATRIX获取。