Reward feedback learning (ReFL) has proven effective for aligning image generation with human preferences. However, its extension to video generation faces significant challenges. Existing video reward models rely on vision-language models designed for pixel-space inputs, confining ReFL optimization to near-complete denoising steps after computationally expensive VAE decoding. This pixel-space approach incurs substantial memory overhead and increased training time, and its late-stage optimization lacks early-stage supervision, refining only visual quality rather than fundamental motion dynamics and structural coherence. In this work, we show that pre-trained video generation models are naturally suited for reward modeling in the noisy latent space, as they are explicitly designed to process noisy latent representations at arbitrary timesteps and inherently preserve temporal information through their sequential modeling capabilities. Accordingly, we propose Process Reward Feedback Learning~(PRFL), a framework that conducts preference optimization entirely in latent space, enabling efficient gradient backpropagation throughout the full denoising chain without VAE decoding. Extensive experiments demonstrate that PRFL significantly improves alignment with human preferences, while achieving substantial reductions in memory consumption and training time compared to RGB ReFL.

翻译：奖励反馈学习（ReFL）已被证明能有效对齐图像生成与人类偏好。然而，其向视频生成的扩展面临重大挑战。现有视频奖励模型依赖于为像素空间输入设计的视觉语言模型，将ReFL优化限制在计算昂贵的VAE解码后接近完成的去噪步骤中。这种像素空间方法带来巨大的内存开销和训练时间增加，且其后期优化缺乏早期监督，仅能优化视觉质量而非基础运动动态与结构连贯性。本工作中，我们证明预训练视频生成模型天然适合在噪声潜在空间中进行奖励建模，因为它们被明确设计为处理任意时间步的噪声潜在表示，并通过其序列建模能力固有地保留时序信息。据此，我们提出过程奖励反馈学习（PRFL），该框架完全在潜在空间中进行偏好优化，无需VAE解码即可在整个去噪链中实现高效梯度反向传播。大量实验表明，与RGB ReFL相比，PRFL在显著提升与人类偏好对齐度的同时，实现了内存消耗和训练时间的大幅降低。