The rapid proliferation of frontier model agents promises significant societal advances but also raises concerns about systemic risks arising from unsafe interactions. Collusion to the disadvantage of others has been identified as a central form of undesirable agent cooperation. The use of information hiding (steganography) in agent communications could render such collusion practically undetectable. This underscores the need for investigations into the possibility of such behaviours emerging and the robustness corresponding countermeasures. To investigate this problem we design two approaches -- a gradient-based reinforcement learning (GBRL) method and an in-context reinforcement learning (ICRL) method -- for reliably eliciting sophisticated LLM-generated linguistic text steganography. We demonstrate, for the first time, that unintended steganographic collusion in LLMs can arise due to mispecified reward incentives during training. Additionally, we find that standard mitigations -- both passive oversight of model outputs and active mitigation through communication paraphrasing -- are not fully effective at preventing this steganographic communication. Our findings imply that (i) emergence of steganographic collusion is a plausible concern that should be monitored and researched, and (ii) preventing emergence may require innovation in mitigation techniques.

翻译：前沿模型智能体的快速扩散有望带来显著的社会进步，但也引发了对其不安全交互可能产生系统性风险的担忧。以损害他人利益为目的的合谋已被确定为不良智能体协作的核心形式。在智能体通信中使用信息隐藏（隐写术）可能使此类合谋在实际中难以被检测。这突显了对此类行为出现的可能性及相应应对措施鲁棒性进行研究的必要性。为探究此问题，我们设计了两种方法——基于梯度的强化学习（GBRL）方法和上下文内强化学习（ICRL）方法——以可靠地诱导出复杂的大语言模型生成的语言文本隐写术。我们首次证明，大语言模型中非预期的隐写合谋可能源于训练期间奖励激励的错误设定。此外，我们发现标准缓解措施——无论是被动监督模型输出，还是通过通信释义进行主动缓解——均无法完全有效阻止这种隐写通信。我们的研究结果表明：（i）隐写合谋的出现是一个值得关注并需持续监测与研究的问题；（ii）防止其出现可能需要缓解技术的创新。