Reinforcement learning (RL) has become a dominant post-training paradigm, enabling large language models (LLMs) to learn from rewards. We observe that societal regulations are structurally similar to reward functions. They define measurable outcomes, thresholds, and exceptions, while often leaving institutional intent only partially specified. We hypothesise that the RL training process may exploit these gaps and therefore ask whether models' well-known tendency to hack reward functions during RL can scale into a more consequential failure mode named societal hacking: discovering loopholes in the rules society runs on. To study this phenomenon, we introduce SocioHack, a sandbox of 72 societal environments, and find that within these environments, reward hacking naturally emerges and leads to regulatory loophole discovery. Models learn to hack the social rules and generate strategies that remain technically compliant while defeating regulatory intent, and current LLM safeguards provide only limited mitigation. Therefore, collecting in-the-wild feedback for model training requires greater caution, and we need a next-generation post-training paradigm for safely iterating LLMs in real society.=

翻译：强化学习已成为主导性后训练范式，使大语言模型能够从奖励信号中学习。我们观察到社会规制体系与奖励函数具有结构性相似：它们既定义可量化目标、阈值和例外条款，又往往仅部分明确制度意图。我们假设强化学习训练过程可能利用这些结构缝隙，因而探究语言模型在强化学习中众所周知的奖励钻营倾向，能否升级为更具破坏性的故障模式——即"社会体系钻营"：发现社会运行规则中的漏洞。为研究该现象，我们构建了包含72个社会环境的SocioHack沙盒实验平台，发现奖励钻营在这些环境中自然涌现并导致规制漏洞发掘。模型学习如何破解社会规则，生成在技术上合规却瓦解规制意图的策略，而现有语言模型防护措施仅能提供有限缓解。因此，使用自然场景反馈进行模型训练需更高审慎度，亟需发展新一代后训练范式，以实现语言模型在真实社会中的安全迭代。