Language models fine-tuned with reinforcement learning typically optimize for task reward, ignoring multi-agent strategic structure. Because these agents condition on natural language game-state descriptions and emit actions through free-form generation, strategic failure modes -- exploiting weaker opponents, coordinating on harmful equilibria, and externalizing costs are inseparable from the language interface itself. We propose Safe Equilibrium Policy Optimization (\sepo{}), a training objective that augments expected payoff with explicit penalties for exploitability, collusion risk, and externality cost. We implement \sepo{} as a reward signal for Group Relative Policy Optimization (GRPO), applied to Gemma~4 E4B-it and Qwen~3.5-4B after supervised fine-tuning (SFT). Evaluated across five strategic domains: Iterated Prisoner's Dilemma, repeated auctions, two negotiation variants, and Kuhn Poker. \sepo{} achieves zero exploit-pool advantage in Kuhn Poker for both models, outperforms the base model on safety in four domains, and corrects the over-cooperative behavior introduced by SFT. In negotiation, \sepo{} achieves a positive-safety outcome and only the positive normalized relative advantage of any negotiation configuration. Ablation experiments confirm that per-rollout exploit computation is necessary: a shared constant penalty cancels in GRPO advantage normalization (constant control-variate property), producing zero gradient. To support further research in strategic safety for agents, we release our \href{https://anonymous.4open.science/r/sepo-2668/README.md}{code} and SFT datasets.

翻译：基于强化学习微调的语言模型通常以优化任务奖励为目标，而忽略了多智能体战略结构。由于这类智能体依赖于自然语言博弈状态描述，并通过自由形式生成输出动作，其战略失败模式——利用弱势对手、协调有害均衡以及外部化成本——与语言接口本身密不可分。我们提出安全均衡策略优化（\sepo{}），该训练目标通过显式惩罚可剥削性、共谋风险及外部性成本来增强期望收益。我们将\sepo{}作为分组相对策略优化（GRPO）的奖励信号实现，应用于监督微调（SFT）后的Gemma~4 E4B-it和Qwen~3.5-4B模型。在五个战略领域（迭代囚徒困境、重复拍卖、两种谈判变体及库恩扑克）的评估中，\sepo{}使两种模型在库恩扑克中实现零剥削池优势，在四个领域的安全指标上超越基础模型，并纠正了SFT引入的过度合作行为。在谈判任务中，\sepo{}实现了正安全结果，且在所有谈判配置中取得唯一正归一化相对优势。消融实验证实逐回合剥削计算的必要性：共享的恒定惩罚在GRPO优势归一化（恒定控制变量特性）中相互抵消，产生零梯度。为支持智能体战略安全的进一步研究，我们开源了\href{https://anonymous.4open.science/r/sepo-2668/README.md}{代码}和SFT数据集。