Faithfully personalizing large language models (LLMs) to align with individual user preferences is a critical but challenging task. While supervised fine-tuning (SFT) quickly reaches a performance plateau, standard reinforcement learning from human feedback (RLHF) also struggles with the nuances of personalization. Scalar-based reward models are prone to reward hacking which leads to verbose and superficially personalized responses. To address these limitations, we propose Critique-Post-Edit, a robust reinforcement learning framework that enables more faithful and controllable personalization. Our framework integrates two key components: (1) a Personalized Generative Reward Model (GRM) that provides multi-dimensional scores and textual critiques to resist reward hacking, and (2) a Critique-Post-Edit mechanism where the policy model revises its own outputs based on these critiques for more targeted and efficient learning. Under a rigorous length-controlled evaluation, our method substantially outperforms standard PPO on personalization benchmarks. Personalized Qwen2.5-7B achieves an average 11\% win-rate improvement, and personalized Qwen2.5-14B model surpasses the performance of GPT-4.1. These results demonstrate a practical path to faithful, efficient, and controllable personalization.

翻译：忠实个性化大型语言模型以符合个体用户偏好是一项关键但极具挑战性的任务。尽管监督微调会迅速达到性能瓶颈，标准的基于人类反馈的强化学习也难以应对个性化的细微差别。基于标量的奖励模型容易陷入奖励破解，导致生成冗长且表面个性化的回复。为克服这些局限，我们提出了批判-后编辑——一个鲁棒的强化学习框架，能够实现更忠实且可控的个性化。该框架整合了两个核心组件：(1) 个性化生成式奖励模型，通过提供多维评分与文本批判来抵御奖励破解；(2) 批判-后编辑机制，使策略模型能够依据这些批判自我修正输出，从而实现更精准高效的学习。在严格的长度控制评估下，我们的方法在个性化基准上显著优于标准PPO算法。个性化Qwen2.5-7B模型平均胜率提升11%，个性化Qwen2.5-14B模型性能甚至超越GPT-4.1。这些结果为实现忠实、高效且可控的个性化提供了切实可行的技术路径。