Aligning large language models (LLMs) with human values is imperative to mitigate potential adverse effects resulting from their misuse. Drawing from the sociological insight that acknowledging all parties' concerns is a key factor in shaping human values, this paper proposes a novel direction to align LLMs by themselves: social scene simulation. To achieve this, we present MATRIX, a novel social scene simulator that emulates realistic scenes around a user's input query, enabling the LLM to take social consequences into account before responding. MATRIX serves as a virtual rehearsal space, akin to a Monopolylogue, where the LLM performs diverse roles related to the query and practice by itself. To inject this alignment, we fine-tune the LLM with MATRIX-simulated data, ensuring adherence to human values without compromising inference speed. We theoretically show that the LLM with MATRIX outperforms Constitutional AI under mild assumptions. Finally, extensive experiments validate that our method outperforms over 10 baselines across 4 benchmarks. As evidenced by 875 user ratings, our tuned 13B-size LLM exceeds GPT-4 in aligning with human values. Our project page is available at https://shuotang123.github.io/MATRIX.

翻译：将大型语言模型（LLMs）与人类价值观对齐对于减轻其滥用可能带来的负面影响至关重要。基于社会学中“承认各方关切是塑造人类价值观的关键因素”这一洞见，本文提出了一种新的方向：通过社会场景模拟让LLMs实现自我对齐。为此，我们提出了MATRIX——一种新颖的社会场景模拟器，它能够模拟用户输入查询周围的真实场景，使LLM在响应前考虑社会后果。MATRIX类似于一个虚拟排练空间（如同独占对话），LLM在其中扮演与查询相关的多种角色，并自行进行演练。为了注入这种对齐，我们使用MATRIX模拟的数据对LLM进行微调，确保其遵循人类价值观且不影响推理速度。我们从理论上证明，在温和假设下，结合MATRIX的LLM优于宪法AI方法。最后，大量实验验证了我们的方法在4个基准测试中优于10余种基线模型。根据875名用户的评分，我们微调的130亿参数级LLM在人类价值观对齐方面超越了GPT-4。项目页面见 https://shuotang123.github.io/MATRIX。