Current language model safety paradigms often fall short in emotionally charged or high-stakes settings, where refusal-only approaches may alienate users and naive compliance can amplify risk. We propose ProSocialAlign, a test-time, parameter-efficient framework that steers generation toward safe, empathetic, and value-aligned responses without retraining the base model. We formalize five human-centered objectives and cast safety as lexicographic constrained generation: first, applying hard constraints to eliminate harmful continuations; then optimizing for prosocial quality within the safe set. Our method combines (i) directional regulation, a harm-mitigation mechanism that subtracts a learned "harm vector" in parameter space, and (ii) preference-aware autoregressive reward modeling trained jointly across attributes with gradient conflict resolution, enabling fine-grained, user-controllable decoding. Empirical evaluations across five safety benchmarks demonstrate state-of-the-art performance, reducing unsafe leakage and boosting alignment to human values, with strong gains across multiple evaluation metrics. ProSocialAlign offers a robust and modular foundation for generating context-sensitive, safe, and human-aligned responses at inference time.

翻译：当前的语言模型安全范式在情感激烈或高风险场景中常显不足，仅依赖拒绝策略可能疏远用户，而盲目顺从则会放大风险。我们提出ProSocialAlign，一种测试时、参数高效的框架，可在无需重新训练基础模型的情况下引导生成安全、共情且价值对齐的响应。我们形式化定义了五项以人为中心的目标，并将安全性构建为词典序约束生成：首先施加硬约束以消除有害续写，随后在安全集合内优化亲社会质量。该方法结合了（i）方向性调节——一种在参数空间中减去习得“危害向量”的减害机制，以及（ii）跨属性联合训练、通过梯度冲突解决实现细粒度用户可控解码的偏好感知自回归奖励建模。在五个安全基准上的实证评估表明，该方法取得了最先进的性能，有效降低了不安全内容泄露并显著提升与人类价值观的对齐度，在多项评估指标上均表现出强劲增益。ProSocialAlign为在推理阶段生成情境敏感、安全且与人类对齐的响应提供了稳健且模块化的基础。