AI red teaming must continually adapt to evolving attackers and defenders. Reinforcement learning offers a promising approach to discovering novel attacks, and co-training methods can produce more robust defenders in tandem. Recent works have demonstrated the efficacy of attacker-defender co-training by applying PPO and DPO, but report that GRPO is unstable in this setting. We introduce AdvGRPO, a co-training framework that makes GRPO viable for joint attacker-defender optimization using dense multi-channel rewards and decoupled advantage normalization. Training progresses through a curriculum from single-turn to closed-loop multi-turn attacks before bootstrapping co-training, where attacker and defender models are updated in alternation. We show that our method can produce highly effective and transferable attacks and that co-trained defenders outperform baselines on safety benchmarks.

翻译：人工智能红队对抗必须持续适应不断演变的攻击者和防御者。强化学习为发现新型攻击提供了有前景的方法，而协同训练方法可同步生成更鲁棒的防御者。近期研究已验证攻击者-防御者协同训练的有效性（通过PPO和DPO实现），但指出GRPO在此场景下存在不稳定性。我们提出AdvGRPO协同训练框架，通过密集多通道奖励与解耦优势归一化，使GRPO能够实现攻击者与防御者的联合优化。训练过程遵循课程式设计：从单轮攻击过渡到闭环多轮攻击，再引导至协同训练阶段（交替更新攻击者与防御者模型）。实验表明，该方法可生成高有效性与可迁移性的攻击，且协同训练后的防御者在安全基准测试中显著优于基线模型。