Recent advancements in Reinforcement Learning with Verifiable Rewards (RLVR) have gained significant attention due to their objective and verifiable reward signals, demonstrating strong performance in reasoning and code generation tasks. However, the potential safety risks associated with RLVR remain underexplored. This paper presents HarmRLVR, the first systematic investigation into the alignment reversibility risk of RLVR. We show that safety alignment can be rapidly reversed using GRPO with merely 64 harmful prompts without responses, causing models to readily comply with harmful instructions. Across five models from Llama, Qwen, and DeepSeek, we empirically demonstrate that RLVR-based attacks elevate the average harmfulness score to 4.94 with an attack success rate of 96.01\%, significantly outperforming harmful fine-tuning while preserving general capabilities. Our findings reveal that RLVR can be efficiently exploited for harmful alignment, posing serious threats to open-source model safety. Please see our code at https://github.com/lyxx2535/HarmRLVR.

翻译：近年来，基于可验证奖励的强化学习（RLVR）因其客观且可验证的奖励信号而受到广泛关注，在推理和代码生成任务中展现出强劲性能。然而，RLVR 的潜在安全风险仍未得到充分探索。本文提出了 HarmRLVR，这是首次对 RLVR 对齐可逆性风险进行的系统性研究。我们证明，仅使用 64 条无响应的有害提示，通过 GRPO 即可快速逆转安全对齐，导致模型轻易遵从有害指令。在 Llama、Qwen 和 DeepSeek 的五个模型上，我们的实验表明，基于 RLVR 的攻击将平均危害评分提升至 4.94，攻击成功率达到 96.01%，显著优于有害微调，同时保留了通用能力。我们的研究揭示，RLVR 可被高效地用于有害对齐，对开源模型安全构成严重威胁。相关代码请见 https://github.com/lyxx2535/HarmRLVR。