Reward models are central to text-to-image post-training, but visual preference is subjective and better represented as a distribution over rubric scores than as a deterministic scalar. Existing scalar, score-token, and pairwise reward models over-compress uncertainty and fine-grained score differences, while reasoning-based generative rewards provide stronger judgments but are costly to deploy and difficult to use as direct optimization signals. We propose Z-Reward, a teacher-student reward modeling framework that decouples reasoning-heavy judgment from efficient reward deployment. The teacher is a large VLM that uses reasoning to infer rubric-aligned score distributions, and is trained with Group-wise Direct Score Optimization (GDSO), which combines policy-gradient rewards from distribution expectations with direct pointwise and pairwise supervision on score distributions and score gaps. The student is trained with Reasoning-Internalized Score Distillation (RISD), which transfers the teacher's reasoning-conditioned score distribution into a compact VLM without requiring explicit reasoning chains at inference time. On our internally annotated evaluation set, the 27B GDSO teacher reaches 89.6% human preference accuracy, outperforming SFT, RewardDance, and GRPO, while the 9B RISD student reaches 88.6%, outperforming the OPD baseline and closely matching the larger teacher. We further show that Z-Reward can serve as a differentiable reward signal for text-to-image optimization, yielding a 41.3% net human-preference improvement over the SFT baseline.

翻译：奖励模型是文本到图像后训练的核心，但视觉偏好具有主观性，更应表示为基于评分标准的分布而非确定性的标量。现有的标量、评分标记和成对奖励模型过度压缩了不确定性和细粒度评分差异，而基于推理的生成奖励虽能提供更强的判断，但部署成本高且难以直接用作优化信号。我们提出Z-Reward，一种教师-学生奖励建模框架，将推理密集型判断与高效奖励部署解耦。教师是一个大型视觉语言模型（VLM），利用推理推断符合评分标准的评分分布，并通过分组直接评分优化（GDSO）进行训练，该方法结合了来自分布期望的策略梯度奖励以及在评分分布和评分差距上的直接逐点和成对监督。学生通过推理内化评分蒸馏（RISD）训练，将教师基于推理条件的评分分布转移到紧凑的VLM中，无需在推理时显式推理链。在我们内部标注的评估集上，270亿参数的GDSO教师达到89.6%的人类偏好准确率，优于SFT、RewardDance和GRPO；而90亿参数的RISD学生达到88.6%，优于OPD基线并与更大的教师模型表现相当。我们进一步表明，Z-Reward可作为可微分的奖励信号用于文本到图像优化，相较于SFT基线实现了41.3%的净人类偏好提升。