Preference optimization for diffusion and flow-matching models relies on reward functions that are both discriminatively robust and computationally efficient. Vision-Language Models (VLMs) have emerged as the primary reward provider, leveraging their rich multimodal priors to guide alignment. However, their computation and memory cost can be substantial, and optimizing a latent diffusion generator through a pixel-space reward introduces a domain mismatch that complicates alignment. In this paper, we propose DiNa-LRM, a diffusion-native latent reward model that formulates preference learning directly on noisy diffusion states. Our method introduces a noise-calibrated Thurstone likelihood with diffusion-noise-dependent uncertainty. DiNa-LRM leverages a pretrained latent diffusion backbone with a timestep-conditioned reward head, and supports inference-time noise ensembling, providing a diffusion-native mechanism for test-time scaling and robust rewarding. Across image alignment benchmarks, DiNa-LRM substantially outperforms existing diffusion-based reward baselines and achieves performance competitive with state-of-the-art VLMs at a fraction of the computational cost. In preference optimization, we demonstrate that DiNa-LRM improves preference optimization dynamics, enabling faster and more resource-efficient model alignment.

翻译：扩散模型与流匹配模型的偏好优化依赖于既具备判别鲁棒性又计算高效的奖励函数。视觉语言模型凭借其丰富的多模态先验知识来指导对齐，已成为主要的奖励提供者。然而，其计算与内存开销可能相当可观，且通过像素空间奖励优化潜在扩散生成器会引入领域不匹配问题，使对齐过程复杂化。本文提出DiNa-LRM，一种扩散原生潜在奖励模型，直接在含噪扩散状态上构建偏好学习。我们的方法引入了一种噪声校准的瑟斯顿似然函数，其不确定性依赖于扩散噪声。DiNa-LRM利用预训练的潜在扩散主干网络配合时间步条件奖励头，并支持推理时噪声集成，提供了一种用于测试时缩放与鲁棒奖励的扩散原生机制。在图像对齐基准测试中，DiNa-LRM显著优于现有基于扩散的奖励基线，并以极小的计算成本实现了与最先进视觉语言模型相竞争的性能。在偏好优化方面，我们证明DiNa-LRM能改善偏好优化动态，实现更快、更资源高效的模型对齐。