Recent breakthroughs in large language models (LLMs) have fundamentally shifted recommender systems from discriminative to generative paradigms, where user behavior modeling is achieved by generating target items conditioned on historical interactions. Yet current generative recommenders still suffer from two core limitations: the lack of high-quality negative modeling and the reliance on implicit rewards. Reinforcement learning with verifiable rewards (RLVR) offers a natural solution by enabling on-policy sampling of harder negatives and grounding optimization in explicit reward signals. However, applying RLVR to generative recommenders remains non-trivial. Its unique generation space often leads to invalid or repetitive items that undermine sampling efficiency, and ranking supervision is sparse since most items receive identical zero rewards. To address these challenges, we propose Reinforced Preference Optimization for Recommendation (ReRe), a reinforcement-based paradigm tailored to LLM-based recommenders, an important direction in generative recommendation. ReRe incorporates constrained beam search to improve sampling efficiency and diversify hard negatives, while augmenting rule-based accuracy rewards with auxiliary ranking rewards for finer-grained supervision. Extensive experiments on three real-world datasets demonstrate that ReRe consistently outperforms both traditional and LLM-based recommenders in ranking performance. Further analysis shows that ReRe not only enhances performance across both base and SFT-initialized models but also generalizes robustly across different backbone families and scales. Beyond empirical gains, we systematically investigate the design space of RLVR in recommendation across generation, sampling strategy, reward modeling, and optimization algorithm, offering insights for future research.

翻译：大型语言模型（LLM）的最新突破从根本上将推荐系统从判别式范式转向生成式范式，其中用户行为建模通过基于历史交互生成目标项目来实现。然而，当前的生成式推荐系统仍面临两个核心局限：缺乏高质量的负样本建模以及对隐式奖励的依赖。带有可验证奖励的强化学习（RLVR）提供了一种自然的解决方案，它能够实现更困难负样本的在线策略采样，并将优化过程建立在显式奖励信号的基础上。然而，将RLVR应用于生成式推荐系统仍非易事。其独特的生成空间常导致无效或重复的项目，从而降低采样效率；同时，由于大多数项目获得相同的零奖励，排序监督信号稀疏。为应对这些挑战，我们提出面向推荐的强化偏好优化（ReRe），这是一种专为基于LLM的推荐系统设计的强化学习范式，代表了生成式推荐的一个重要方向。ReRe引入约束束搜索以提高采样效率并多样化困难负样本，同时通过辅助排序奖励增强基于规则的准确性奖励，以实现更细粒度的监督。在三个真实世界数据集上的大量实验表明，ReRe在排序性能上持续优于传统推荐系统及基于LLM的推荐系统。进一步分析显示，ReRe不仅提升了基础模型和SFT初始化模型的性能，还能在不同骨干网络架构和规模上实现稳健的泛化。除实证收益外，我们系统性地探索了RLVR在推荐系统中的设计空间，涵盖生成策略、采样方法、奖励建模和优化算法，为未来研究提供了重要见解。