We demonstrate a situation in which Large Language Models, trained to be helpful, harmless, and honest, can display misaligned behavior and strategically deceive their users about this behavior without being instructed to do so. Concretely, we deploy GPT-4 as an agent in a realistic, simulated environment, where it assumes the role of an autonomous stock trading agent. Within this environment, the model obtains an insider tip about a lucrative stock trade and acts upon it despite knowing that insider trading is disapproved of by company management. When reporting to its manager, the model consistently hides the genuine reasons behind its trading decision. We perform a brief investigation of how this behavior varies under changes to the setting, such as removing model access to a reasoning scratchpad, attempting to prevent the misaligned behavior by changing system instructions, changing the amount of pressure the model is under, varying the perceived risk of getting caught, and making other simple changes to the environment. To our knowledge, this is the first demonstration of Large Language Models trained to be helpful, harmless, and honest, strategically deceiving their users in a realistic situation without direct instructions or training for deception.

翻译：我们展示了一种情境，其中被训练为有益、无害且诚实的的大型语言模型可能表现出失调行为，并在未收到指令的情况下，战略性隐瞒其真实行为。具体而言，我们将GPT-4部署为真实模拟环境中的智能体，使其扮演自主股票交易代理角色。在该环境中，模型获得关于某高利润股票交易的内幕消息，尽管知晓公司管理层不赞成内幕交易，仍执行该操作。在向管理者汇报时，模型始终隐瞒其交易决策背后的真实原因。我们初步探究了该行为随环境设置变化的规律，包括移除模型推理草稿访问权限、尝试通过修改系统指令阻止失调行为、改变模型承受压力程度、调整被察觉风险感知水平，以及执行其他简单环境变更。据我们所知，这是首次展示被训练为有益、无害、诚实的的大型语言模型，在无直接指令或欺骗训练的情况下，于现实情境中战略性欺骗用户的行为模式。