While reasoning models have achieved remarkable success in complex reasoning tasks, their increasing power necessitates stringent safety measures. For safety alignment, the core challenge lies in the inherent trade-off between safety and utility. However, prevailing alignment strategies typically construct CoT training data with explicit safety rules via context distillation. This approach inadvertently limits reasoning capabilities by creating a rigid association between rule memorization and refusal. To mitigate the safety-utility trade-off, we propose the Adaptive Safe Context Learning (ASCL) framework to improve the reasoning given proper context. ASCL formulates safety alignment as a multi-turn tool-use process, empowering the model to autonomously decide when to consult safety rules and how to generate the ongoing reasoning. Furthermore, to counteract the preference for rule consultation during RL, we introduce Inverse Frequency Policy Optimization (IFPO) to rebalance advantage estimates. By decoupling rule retrieval and subsequent reasoning, our method achieves higher overall performance compared to baselines.

翻译：尽管推理模型在复杂推理任务中取得了显著成功，但其日益增强的能力需要严格的安全措施。对于安全对齐，核心挑战在于安全性与实用性之间固有的权衡。然而，主流对齐策略通常通过上下文蒸馏构建包含明确安全规则的思维链训练数据。这种方法无意中通过建立规则记忆与拒绝之间的僵化关联，限制了推理能力。为缓解安全性与实用性的权衡，我们提出了自适应安全上下文学习框架，以在给定适当上下文的情况下改进推理。ASCL将安全对齐形式化为多轮工具使用过程，使模型能够自主决定何时参考安全规则以及如何进行持续推理。此外，为抵消强化学习期间对规则咨询的偏好，我们引入了逆频率策略优化来重新平衡优势估计。通过解耦规则检索与后续推理，我们的方法相较于基线模型实现了更高的整体性能。