Alignment is vital for safely deploying large language models (LLMs). Existing techniques are either reward-based (train a reward model on preference pairs and optimize with reinforcement learning) or reward-free (directly fine-tune on ranked outputs). Recent research shows that well-tuned reward-based pipelines remain robust, and single-response demonstrations can outperform pairwise preference data. However, two challenges persist: (1) imbalanced safety datasets that overrepresent common hazards while neglecting long-tail threats; and (2) static reward models that ignore task difficulty, limiting optimization efficiency and attainable gains. We propose DR-IRL (Dynamically adjusting Rewards through Inverse Reinforcement Learning). We first train category-specific reward models using a balanced safety dataset covering seven harmful categories via IRL. Then we enhance Group Relative Policy Optimization (GRPO) by introducing dynamic reward scaling--adjusting rewards by task difficulty--data-level hardness by text encoder cosine similarity, model-level responsiveness by reward gaps. Extensive experiments across various benchmarks and LLMs demonstrate that DR-IRL outperforms all baseline methods in safety alignment while maintaining usefulness.

翻译：对齐对于安全部署大语言模型至关重要。现有技术主要分为基于奖励的方法（在偏好对上训练奖励模型，并通过强化学习进行优化）和免奖励的方法（直接在排序输出上进行微调）。近期研究表明，精心调优的基于奖励的流程依然稳健，且单响应演示数据可能优于成对偏好数据。然而，两个挑战依然存在：（1）安全数据集不平衡，过度代表常见危害而忽视长尾威胁；（2）静态奖励模型忽略任务难度，限制了优化效率和可达到的收益。我们提出DR-IRL（通过逆强化学习动态调整奖励）。我们首先使用覆盖七类有害类别的平衡安全数据集，通过逆强化学习训练特定类别的奖励模型。然后，我们通过引入动态奖励缩放来增强组相对策略优化——根据任务难度调整奖励：通过文本编码器余弦相似度衡量数据级难度，通过奖励差距衡量模型级响应度。在多种基准测试和大语言模型上进行的大量实验表明，DR-IRL在安全对齐方面优于所有基线方法，同时保持了实用性。